Big Games, Small Screens
Developing 3D games for mobile devices is full of challenges, but the rich, evolving toolset enables some stunning results.
MARK CALLOW, HI CORPORATION
PAUL BEARDOW, ECCOSPHERE
DAVID BRITTAIN, SUPERSCAPE
One thing that becomes immediately apparent when creating and distributing mobile 3D games is that there are fundamental differences between the cellphone market and the more traditional games markets, such as consoles and handheld gaming devices. The most striking of these are the number of delivery platforms; the severe constraints of the devices, including small screens whose orientation can be changed; limited input controls; the need to deal with other tasks; the nonphysical delivery mechanism; and the variations in handset performance and input capability.
Outside of the mobile market, developers had to target only two or three devices and deliver games on high-capacity media; in the mobile market they have to consider tens of devices per operator and package their games to fit in a compact download. The number and types of devices are also constantly changing; in a 12-month development cycle many new handsets can emerge. In addition, console development focus has been about length of play and numbers of levels, rather than the short bursts of intense activity that typify today’s mobile games. Furthermore, development time and budget are usually limited by low retail prices in the $5-10 range.
Dealing with multiple platforms and the need to manage a reasonably sized development team leads to placing great emphasis on up-front planning and design of a game. Right from the start, the developer needs to consider what the target platforms are, will be, and could be. If a reasonable return on development investment is required by earning revenue from as many users as possible, then the developer will most likely look at Java games running on today’s mass-market handsets, as this gives them access to about 60 percent of the market. The games will use embedded Java APIs such as M3G (Mobile 3D Graphics API for Java, a.k.a. JSR-184) and MascotCapsule. Given a typical development cycle of 9 to 12 months including testing and operator certification, they will also need to make an educated guess about what new handsets will be arriving over the next year and allow for introduction of new technologies, such as hardware acceleration, which could completely change the look of the game.
The game design must consider the orientation of the screens on the target devices. Must both landscape and portrait be supported? Can the orientation be changed on the fly? Handsets whose screen orientation can be changed on the fly are increasingly common, driven by wide-screen TV viewing, and they present some unique opportunities. For example, a labyrinth-style game might switch between the normal first-person view and a map view when the screen is rotated.
Another important design consideration is the range of input capabilities across the target platforms. Keypad and control layouts vary. Some will map better than others to the game action. Changing the screen orientation may make the keypad and controls harder to use. Diagonal motion and simultaneous button pushes may be possible on some platforms, but impossible on others. Keyless touch screens, while still relatively rare, must also be considered. Solutions can involve multiple difficulty levels—defaulting to an easier level where, for example, diagonal input is not possible—putting the same feature on multiple buttons, or creating user-customizable controls.
In a racing game, for example, the user might be able to use either the navigation controller or various keys on the 12-key pad to control acceleration, braking, and steering. The user might also be able to choose from a selection of key layouts for these and other controls such as gear changing. When running on a handset where diagonal input is not possible, the game may select an easier level where the steering is less sensitive and braking reacts faster to allow for possible delay in changing from steering to braking.
Once the range of potential performance of the target platforms has been estimated or, better yet, measured, and the input capabilities planned for, then the developer has to start considering delivery of the games. Are the target handsets all Java? Are some Symbian? Brew perhaps, too? On top of that, download size of the game is of paramount importance. Size limits are governed mostly by network performance and robustness characteristics, but in many cases operators or handsets impose concrete limits. Today Java networks, including most 3G ones, support only around 300- to 500-KB OTA (over-the-air) downloads. Brew devices, on the other hand, can be 1 MB and more. Symbian games may be delivered by other means and so could be multiple megabytes if required, but no attractive distribution channel exists outside of OTA downloads.
Careful up-front planning is also important to ensure that the costs of porting and testing all these ports don’t spiral out of control. It’s important to make sure that there are as few changes as possible between individual packages. One way to do this is to establish families of handsets with similar capabilities; another is to use porting technologies that abstract game code from handset specifics. A small number of commercial porting platforms are available. Either way, minimizing the differences means that bug fixes or modifications can be made to the masters rather than the individual ports, which would be much more expensive. The best approach is to combine the two techniques: having a porting platform that fixes known device bugs allows developers to focus on creating good games instead of worrying about how to make a game work on a specific phone. Device families help in porting, QA, and distribution, as they reduce the number of SKUs to develop, QA, manage, and deliver. The management and execution of the mobile game delivery process is complex, and to a large degree cost scales with SKU count: fewer SKUs equals lower-cost game delivery.
Platform variety issues can be mitigated by standardization of APIs. OpenGL ES (embedded systems) and M3G (see the accompanying sidebars for a more detailed discussion of these two APIs) have done much to ease the pain of game developers. Varying handset performance, however, will always mean that game designs need to be scalable for widespread handset coverage, even more so with the 3D accelerated devices that are in the market now and, undoubtedly, with even faster ones emerging over the next year. In tandem with scalability, games also need to address portability so that sections of code do not need to be rewritten time and time again.
Designing For Scalability
Designing a scalable game requires the team to consider multiple sets of game assets or multiple display-drawing mechanisms from the outset. The same game logic may apply, but the models, textures, and order of drawing may be altered, removed, or substituted to allow games to download to and execute on the desired handset. This is where a retained-mode1 API such as M3G has a distinct advantage: the basic drawing code can remain the same if it is a scene-tree render, rather than being constructed from many low-level calls. The tradeoff may be fine control of the order of drawing of objects, but when considering porting to many handsets, this may be an acceptable tradeoff. An additional advantage of an API such as M3G that is embedded in a handset is that it can be well tuned for any 3D hardware that may be installed in the handset. The handset maker and 3D API vendor can afford to invest more time tuning the API for the hardware than a game developer could invest in tuning an individual game, given the number of handsets.
On nonaccelerated handsets it is possible to counter lower frame rates by mixing 2D and 3D graphics calls, so that, for example, a 3D character could be drawn over a 2D bitmap background. Java ME (Micro Edition), for example, already has 2D APIs that can be used in a complementary fashion to 3D. This works best on nonaccelerated platforms. On accelerated platforms it is usually more efficient to keep screen updates all 3D-based to allow optimal code paths through the low-level renderer and hardware. In fact, it is generally the case that if you design your game assets with hardware acceleration in mind, they will also run well on software renderers; the converse is not true.
One area to be aware of is text and internationalization. If the developers decide to use bitmaps for fonts, then they need to ensure that the bitmaps’ widths and heights are powers of two for straightforward use as textures on a hardware accelerator. They also need to make sure that the download has enough space for all the languages required.
DESIGNING FOR MULTIPLAYER GAMING
Because mobile phones are connected devices with strong community development possibilities, they make excellent candidates for multiplayer gaming. These types of games have a critical barrier to cross in order to be successful, however, and that is developing an adequate player base. When players want a game, they have to be able to find opponents or teammates. Often implementations require the user to wait for long periods while a multiplayer session is initiated and then wait in a lobby for other players to become available. A good game design should allow the server, after a short period of time, to fall back to bots (AI-based or predefined server-side players). In many games the user might never be able to tell the difference between these and real players.
This leads to another challenge for online gaming: how do you add personality to your online opponents? In PC and console gaming it is relatively simple to allow players to multitask—so that they can both play the game and chat either through text or voice. On constrained mobile devices this is much more challenging because the user interface and keypad are so limited. Predefined “taunts” can help simplify the message-sending process, but given that these are audio-enabled devices, using that ability has to be part of the long-term solution. The technology currently available through device APIs does not allow this level of access for today’s game developer.
Of course, the biggest headaches with multiplayer gaming are still the bugbears of latency and device variability, so games requiring fast response, such as first-person shooters or racing games, are still limited to high-end 3G handsets. Numerous solutions exist for console games, but in the mobile arena the challenge is much bigger since the range between best and worst is so much greater. Furthermore, it is not clear whether a sufficient volume of devices is in the market yet, or whether consumers are prepared to pay more for the experience. Adding online features to a mobile title considerably increases costs throughout the process, from development to porting and QA.
Multiplayer and community games are a good match for the mobile audience. A well-thought-out game that does not make itself exclusive to 3G and high-end devices could easily be the killer app for mobile gaming.
The Development Process
Development of 2D mobile games is already well established, and 3D development in principle is not substantially different—it is, after all, just a different way of drawing the game. Both have code and assets and require packaging, testing, and certification for delivery to the end user. The difference is that 3D development can have more complex asset creation and potentially more complex game code, although this is not necessarily the case and many 3D games in reality can be reduced in concept, and therefore code, to a 2D equivalent (many 3D racing games are actually 2D games with 3D models).
The basic assets and code needed for a Java 3D mobile game are shown in figure 1. These assets extend those for 2D and add 3D geometry, textures for object appearance, lighting and camera parameters, and perhaps higher-level constructs such as animation tracks and bones. In M3G, for example, all those assets, as well as morphing data, groupings, layers, and 2D sprites, can be stored in one or more scene trees held as data files. Using data files to store 3D assets has a huge advantage over constructing assets at runtime in code because it enables artists and designers to create the assets using familiar design tools.
One of the most popular tools for creating 3D models in the games industry is 3D Studio Max. Users can typically heavily customize tools of this nature through plug-ins, a number of which are available for 3ds Max enabling exporting M3G content. Artists can then create and preview 3D models in an interactive visual environment and at the touch of a button export to an M3G file that can be previewed in an emulator, as illustrated in figures 2 and 3. The coder then takes over and writes code to load and manipulate this content on a mobile device.
Using multiple data files allows designers to work on individual elements of a scene, as well as allowing scenes to be dynamically constructed within the game. That makes it possible for one designer to build the levels of a game while another models the characters. When the game is executed, the chosen characters can be inserted programmatically into the scene, as play requires.
Developers will find great disparity in sound quality between different handsets, ranging from low-quality mono to the high-quality 3D positional audio found on some recent models. It is important to create sound assets at the highest possible quality. It is far easier to reduce the quality of the assets than increase it when tailoring for a particular handset and download size.
In the console world the game is the only application; in the mobile world the game has to be well behaved when a call or text message arrives. It must not cause problems with the normal use of the phone or interfere with other background applications. This is a new discipline to many developers, and dealing with it may lead to some bugs. One solution is to create a framework that copes with these issues for each handset or family of handsets and embed the game into this framework. Creating this framework will require more development time for the first game on a handset family, but from the second sibling terminal or game, the developer can focus on the programming.
An incoming call on most platforms will cause a pause event to be sent to the game, which must suspend game time, stop updating the game world, and stop rendering. Usually a worker thread performs these update tasks so the game simply records the current time and kills the worker thread. The game must also release any shared resource such as the 3D graphics rendering context in M3G. On more restricted platforms, the application may need to reduce memory use by deleting some of its assets. In this case, the game records enough information to allow it to re-create the assets in the same state when it eventually receives a resume event.
Programming for constrained devices such as handsets requires a great deal of discipline in managing memory usage. This is especially true in Java handsets where memory allocation or releasing can cause unexpected slowdowns in games because of garbage collection. Sometimes it’s hard to see where the allocation is happening, but in general it’s important for performance reasons not to allocate any memory at all in the main game loop. Equally important is managing overall memory use, especially when using bitmaps or sounds. Loading resources when needed and then deleting them is critical, and this must be done so as not to cause fragmentation of the Java heap. Often the best solution is for game developers to write their own memory manager code so that the allocation and freeing of all resources is tightly controlled. This is not as easy as when writing native code, since you are stuck with using the Java VM’s memory allocator, but you can write higher-level resource management code that controls when images, text, and other data get loaded and unloaded.
For example, a simple game may have various sets of assets: (1) start-up sequence images including splash screens; (2) the menu system graphics; and (3) graphics used in two game levels. Set 1 will be loaded once and then permanently discarded. Set 2 may be needed regularly as the user enters and exits different levels of the game. Set 3 data for each level is likely to be large and so it may not be possible to keep it all in memory. To manage this memory, you must load all data from set 1 in one contiguous chunk of memory so that when discarded, a contiguous block is freed back to the system. Data in set 2 should be loaded once and remain in memory at all times; otherwise, continuously loading and freeing objects can cause problems on many handsets. Again, this data should be loaded consecutively to avoid fragmentation. Finally, the level data in set 3 should be loaded and then freed in one operation at the beginning and end of the level. If at all possible, images or other data should be shared between the menu and game level to reduce overall heap requirements.
It is particularly important in 3D game development to keep checking on real hardware and not develop on emulators exclusively. Emulators tend to have little in common performance-wise with the actual devices and may even use different Java virtual machines and 3D API implementations. Java ME has no cycle-accurate emulators, and developing them would be a challenging task as each device has performance characteristics based on so many different factors. Developers need to think about reference handsets—and they should be reasonable devices, not the fastest ones; otherwise, they will be faced with major porting issues at the end of game development. Always remember during the process that the game has to work on multiple platforms if it is intended to reach the widest possible audience.
Porting, Testing, and Packaging
At the end of its initial development, the game has to be ported to the target handsets. Although a developer could choose to limit the game ports to those handsets that are most powerful or have a certain screen size, in general the game should run on as many handsets as possible.
The first challenge is therefore how to create and manage the code for all of the handsets for which builds are needed. Without good tools, libraries, and careful game design, it is very easy either for the number of source-code copies to explode or for the source code to become completely unreadable with preprocessor statements. The most common solution is the porting platform mentioned earlier. This platform abstracts the device differences, presenting a common API across devices, and fixes all known device issues. The game source code can then be written to be much more device agnostic.
The biggest problems faced in porting are defects in handset software, small heap sizes, proprietary APIs, and the biggest of all, performance. Some of the most popular handsets for games are also the cheapest (and therefore the most restricted) devices, and operators often require that games be ported to these models.
Testing is another area that is often overlooked. Delivering games that must not crash is obvious, but equally as important, they must not cause problems with the normal use of the phone. They should not run the battery down unnecessarily, block important information on the screen, or use up so much processor time that the phone becomes unresponsive.
Once a game is QA’d and ready to go, the developer must get it ready for operator testing. In general, this means packaging it with the right details in the JAD (Java application descriptor) files, making sure its splash screens and menu structure comply with the operator’s style guide, and finally that it is the right size for download. Games that pass operator testing are digitally signed so they can be installed via OTA downloads.
Maintaining the ports across new handsets is also important. For a successful game the developer needs to keep the porting rolling for the lifetime of the game to ensure access to the latest and greatest handsets.
Publishing and Distribution
In the mobile market, OTA distribution is the only viable method. Any type of distribution involving a PC will encounter various difficulties—customers having trouble finding you, billing the small amounts of money involved, and users not being able to transfer games from their PCs to their handsets. Also, downloading via a PC is not a casual undertaking—it cannot be done on a whim while waiting for an appointment.
With the exception of a few high-end devices, distribution via cartridges or optical media is out of the question.
There are three OTA distribution channels: on deck, off deck, and off portal. The deck is a set of links to game downloads that the operator maintains. These are accessed by, for example, a More Games button in a handset’s game menu. In on-deck distribution the operator provides billing services, and purchases appear on the user’s monthly phone bill. In return for this and for providing space on the deck, the operator takes a percentage of the purchase price. In Japan the operators usually take around 6 percent. In Europe they have been known to demand as much as 90 percent, with 50 to 60 percent being the norm. In the United States 40 to 50 percent is typical. In return, operators may provide help with marketing, but not always.
Deck placement and operator relationship has become the key factor in the success or failure of game publishers and developers. With the big guns entering the market, a new developer is highly unlikely to get content listed without big brands or an existing contracted publisher. Deck managers these days typically are not accepting new publishers, even though they are always looking for new content to keep revenues flowing and they publish “What’s New” lists as often as twice monthly. The small staffs simply can’t cope with a large number of individual submissions.
Off deck is a harder-to-find channel within the operator’s portal (e.g., AT&T Wireless’s Beyond Media Mall). Typically, users find games by using a search function in the operator’s portal. The carrier provides billing services but the percentage cut is smaller and, more importantly for smaller publishers, the barriers to entry are not nearly as high as for on-deck distribution.
Off-portal distribution is completely independent of the operator. The developer maintains a Web site or uses an off-portal publisher. This is the most challenging channel in terms of customers finding the developer, especially since only 4 percent of mobile-phone users have managed to purchase a game from the game deck. Because typical game prices are too low for credit card transactions, subscription plans must be offered. The only other revenue mechanisms are premium SMS messages or advertiser funding.
The ability to get games from URLs outside the walled garden of the deck is key to the longevity of any publisher because a top 10 game deck won’t sustain a large publisher community. In response to this, several off-portal publishing services have sprung up on the Web, including one where all games are advertiser funded. The introduction of Wi-Fi handsets will also extend the possibilities, as will links to Wi-Fi/IR-enabled consoles. Traditional game publishers will start to look at cross-marketing between mobile and console titles. Of course, the jury is still out on whether console brands transfer to the mobile market. Indications so far show that the casual titles dominate.
A mobile game developer starting in the business today will need to build a good relationship with a publisher or have a good viral marketing plan to overcome the difficulties of off-portal distribution.
Largest Potential Games Audience
The mobile gaming space is clearly a very different market from the console and PC markets—in the nature of the devices, the audience, and the route to market. Successful game developers and publishers must be willing to adapt to these differences; attempting to transport console or PC content without addressing the unique features of this new environment will lead to failure. Prospective developers and publishers must ask themselves these questions: Can I find a route to market? Can my game be adapted for handsets’ features and constraints? And probably most importantly, is my game suitable for the mobile audience?
Never assume that a good console game will become a successful mobile game. The way mobile games are played is far more akin to arcade-style gaming, where short bursts of rapid entertainment are required. Don’t come to market with a multiplayer title unless you have a solid mechanism for establishing a critical mass of players. Above all, make sure that you are as aware as possible of the game design, development, and porting tasks to make sure that your game can get in front of as many players as possible—and make it a rewarding experience.
The mobile market has the largest potential games audience ever. More than 700 million new handsets ship each year, and the majority of those are enabled for gaming in some form. That is many times the console space and encompasses a much wider range of consumers and appetites for entertainment.
To address this market, developers and publishers may need to think differently about what constitutes a “game” and put more thought into end users’ “playing.” That may mean a trend to less-traditional games and more ideas that match the multiple but very large audience niches that will exist.
- A style of 3D graphics programming where a persistent representation of the graphical objects, their spatial relationships, and their appearance together with the viewing position is held in a scene graph in memory. The program draws the entire scene with a single library call, avoiding low-level work in managing, culling, and rendering the data.
MARK CALLOW is chief architect at HI Corporation, Tokyo, where he designs 3D graphics engines for mobile phones. He is a member of the JSR-297 (M3G 2.0) Expert Group and contributes to several Khronos Organization working groups, including OpenGL ES. Callow was an independent consultant for a number of years before joining HI and prior to that worked for many years at Silicon Graphics.
PAUL BEARDOW is CTO of Eccosphere, developing the next generation of mobile social networking. Prior to that he was CTO of Superscape, U.K., where he led the development of 3D software technology for handsets and game software development kits, as well as setting up the games development studio that created 3D titles for Disney, 20th Century Fox, and Sony Pictures. He was a member of the initial expert group for M3G 1.0 and represented Superscape at the Khronos Organization.
DAVID BRITTAIN is vice president of technology at Superscape, U.K. He was part of the team that created Superscape’s M3G implementation and now leads the development of Superscape’s mobile game development tools.
Refining OpenGL for Embedded Systems
Kari Pulli and Jani Vaarala, Nokia, and Ville Miettinen
The early mobile 3D engines were all proprietary, and their rendering pipelines were implemented in software running on general-purpose CPUs. In 2002, to provide a baseline for portable 3D hardware acceleration, the Khronos Group began to design a 3D standard for mobile devices. The idea was to take OpenGL, the most widely deployed 3D API, as a starting point to produce a leaner and cleaner version: OpenGL ES (embedded systems).
Simplification involved removing rarely used features (e.g., feedback and selection rendering modes, and functionality that’s mostly syntactic sugar, such as the GL utility library). It also meant removing redundancy. For example, OpenGL provides a number of different ways of defining the rendering primitives. OpenGL ES provides only one: all vertex data is provided in arrays, and this results in a simpler implementation and faster execution. Also the set of supported geometric primitives is limited to points, lines, and triangles.
OpenGL ES retains most of the transformation and lighting pipeline of OpenGL. Only the back buffer is accessible for drawing and reading, however, and only the RGBA color mode (no indexed color mode) is supported. Features easily emulated using texture mapping (bitmaps and stippling of lines and polygons) were dropped; all key 2D texture-mapping features were retained.
The OpenGL ES graphics pipeline shown in the adjacent figure has only a couple of well-defined points for providing input, and it has few outputs other than the values rendered into the frame buffer. Therefore, functionality in the back end, which cannot be easily replaced by the application programmer, is much more costly to remove than functionality in the front end of the pipeline. As a consequence, OpenGL ES supports almost all of the back-end functionality of OpenGL 1.3. For example, the blending modes that dictate how new fragments should be mixed with existing pixels in the frame buffer were retained in full, as were the various tests for determining whether a pixel should be drawn.
Today only the highest-end mobile devices have FPUs (floating-point units); in 2002 practically no device did. Without an FPU, a device must use software emulation to perform all floating-point arithmetic; this slows down computations by at least an order of magnitude. OpenGL ES addresses this issue by replacing double-precision floats with single-precision floats and providing an integer (fixed-point) variant for each API function. The main and most widely adopted profile of OpenGL ES, the Common profile, supports floating-point functions, whereas the Common-Lite profile, intended for ultra low-end devices, does not.
Two newer versions of OpenGL ES have appeared since the initial specification. Version 1.1 is backwards-compatible. It guarantees the existence of at least two texturing units and provides texture combiners for more versatile combining of multiple texture maps. For example, with dot3 bump mapping, one can add apparent surface roughness to the 3D models. This means using fewer triangles for the models while providing the appearance of a highly detailed surface that participates in illumination effects. Other new features include distance-attenuated point sprites that can be used for particle effects (e.g., fire or rain). Buffer objects allow the geometry data to be stored in the memory of the graphics hardware. This leads to reduced data transfers and higher rendering performance. Optional extensions are available: one for efficient pasting of bitmaps onto the frame buffer and one for smooth character animation using a technique commonly known as skinning.
OpenGL ES 2.0 is not backwards-compatible with the previous versions. To keep the API simple, functionality that can be expressed using programmable shaders (the white boxes in the figure) completely replace the fixed functionality within that part of the pipeline. The vertex transformation and lighting modules have been replaced by a programmable vertex shader stage, while texture mapping, summing of various color components, and fog calculations are now done in a fragment shader. The shaders are written in GLSL ES, a high-level programming language similar to C. The 2.0 hardware is expected to support 1.1, providing application-level backwards compatibility.
OpenGL ES was designed to be an operating system-independent 3D rendering API that is totally independent of surface and context management. The optional EGL API manages and abstracts graphics system resources. Later EGL versions allow combining OpenGL ES with APIs such as OpenVG, which is targeted for scalable 2D vector graphics.
Almost all hardware designs targeted for mobile devices now support OpenGL ES. It can be used to implement higher-level libraries—for example, M3G accesses graphics hardware via OpenGL ES.
KARI PULLI is a research fellow at Nokia, where he has led many of Nokia’s graphics research and standardization efforts. Currently he heads a research team at Nokia Research Center, Palo Alto. He has a Ph.D. from the University of Washington and has worked at the University of Oulu, Stanford University, and MIT.
JANI VAARALA is a graphics architect at Nokia. He has been involved with OpenGL ES standardization and headed a project that developed a software-based OpenGL ES engine and adapted EGL for Symbian. He started on 3D graphics in the early ’90s working on an Amiga, on which he developed several award-winning graphics demos. He is a member of ACM SIGGRAPH, Eurographics, and the Khronos Group.
VILLE MIETTINEN was the CTO and cofounder of Hybrid Graphics, which was merged into NVIDIA. He now works on his own projects. During the last decade he has been involved in the design and implementation of numerous software products in the games and 3D graphics industries. His research interests include dynamic code generation and software rasterizers. He is a member of ACM SIGGRAPH, the Khronos Group, and the JSR-184 expert group.
M3G: Bringing 3D Graphics to Mobile Java
Tomi Aarnio, Kimmo Roimela, and Kari Pulli, Nokia
M3G (Mobile 3D Graphics API for Java, a.k.a. JSR-184) is an easy-to-use yet powerful 3D graphics API for mobile Java. It is primarily a retained-mode API: a model of the 3D scene is maintained inside the API, and individual objects and rendering assets can be inserted and manipulated via the API functions. The low-level rendering model is based on OpenGL ES, with the higher-level functionality layered on top.
Compared with 2D bitmap graphics, 3D allows the games to pack larger numbers of smoother and richer animations into the same amount of memory. It also improves the sense of depth and perspective and enables cinematic camera controls, as illustrated in the adjacent figure. Most of the processing time in an interactive 3D game is usually spent in the core 3D routines that execute in optimized native code or use dedicated 3D hardware. Implementing 3D rendering in Java alone would be prohibitively slow, which is why M3G was designed.
At the lowest level, M3G deals with concepts similar to those in OpenGL ES: vertex buffers, textures, light sources, materials, and transformation matrices. These are the building blocks for higher-level objects and scene graphs. Vertex and index buffers can be combined into Mesh objects; textures, materials, and other rendering parameters form Appearance objects for shading; Group nodes allow logical grouping and hierarchic transformation of scene elements.
The higher-level features, enabled by the scene-graph approach, include functionality commonly required in games and other interactive 3D graphics applications. The aim in designing M3G was to build in common functionality that most applications will need in any case, without being too application-specific. This reduces application size and improves developer productivity. It also improves overall application performance, because raising the abstraction level allows M3G to batch-process entire 3D scenes in native code.
The built-in animation engine is based on keyframe animation. Keyframe tracks can be constructed for linear or spline interpolation of arbitrary parameters (scalar, vector, orientation, color) and attached to almost any property of any object. M3G also supports vertex deformation through morphing and skinning. Separate AnimationController objects control the speed and timing of animation. The whole scene graph can then be animated with a single function call, and the application can freely change any of the keyframes or animation parameters between calls for dynamic effects.
The compact, binary M3G file format encourages the separation of code and content so that they can be independently modified without breaking the complete application. The format is flexible enough to store anything from complete 3D animations and scene graphs down to individual objects or their components. Each object in an M3G file can be given a unique ID number, and that number used to find the object after loading a complex file into the application. This enables the graphics designer to tag objects such as the head, feet, and arms of a character model for easy access by the application code. Additional user data can be associated with each object in the file format to allow editing application-specific data directly in the content tools.
For added flexibility, M3G also supports OpenGL ES-like immediate-mode rendering, where the application merely sends rendering primitives to the API without any associated semantic structure. This can be useful for special effects or whenever the application just wants maximum control over the rendering process. The same data objects are used for both retained- and immediate-mode rendering, so the two can be interleaved. The retained mode, however, can usually provide better performance because of reduced Java overhead and better batching of operations inside the API.
In essence, M3G is a powerful graphics server for the Java application, handling the rendering efficiently while only the application code needs to run in the slower Java. By providing features beyond rendering, M3G also relieves some of the burden on the application code.
M3G version 2.0 will provide programmable shaders and other OpenGL ES 2.0 features for high-end devices, as well as enhanced traditional rendering for the mass market. The animation, scene graph, and utility parts of the API will also be extended, allowing applications to shift more and more of their heavy computations from Java code to native code. The first M3G 2.0 devices will reach the market in 2009; by that time more than a billion M3G-enabled devices probably will have been shipped.
TOMI AARNIO is the specification lead and editor of the M3G standard (JSR-184, 297). As a senior research engineer at Nokia, he has been involved in designing and implementing several mobile graphics engines, most recently heading the implementation of M3G. His research interests include realtime rendering, data compression, and hardware abstraction. He is a member of ACM SIGGRAPH, Eurographics, and the Khronos Group.
KIMMO ROIMELA is an associate editor of the M3G specification. As a senior research engineer at Nokia, he has also been actively involved in implementing the specification. He has an M.Sc. from Tampere University of Technology and is based in the Nokia Research Center in Tampere, Finland.
KARI PULLI is a research fellow at Nokia, where he has led many of Nokia’s graphics research and standardization efforts. Currently he heads a research team at Nokia Research Center, Palo Alto. He has a Ph.D. from the University of Washington and has worked at the University of Oulu, Stanford University, and MIT.
Originally published in Queue vol. 5, no. 7—
see this item in the ACM Digital Library
MARK CALLOW is chief architect at HI Corporation, Tokyo, where he designs 3D graphics engines for mobile phones. He is a member of the JSR-297 (M3G 2.0) Expert Group and contributes to several Khronos Organization working groups, including OpenGL ES. Callow was an independent consultant for a number of years before joining HI and prior to that worked for many years at Silicon Graphics.For additional information see the ACM Digital Library Author Page for: Mark Callow