Building MMO Virtual Worlds (part 2)

» Scaling the Virtual World

One of the challenges in creating large multi-user virtual worlds is creating a scaleable server side infrastructure which allows the system to keep up with the increasing traffic and popularity of the application. These projects usually start relatively small, offering free memberships to users so they can evaluate it and decide if they want to become registered members. In this phase the number of concurrent users hitting the application typically ranges from a few hundreds to a few thousands, so the investment in hardware resources is not particularly onerous: a couple of high-specced machines would handle the load without problems.

As soon as the word is spread and banner campaigns are launched, more and more users will sign up for both free and paid memberships and we may find out that the current hardware setup is lacking horse-power.

There are various options for building a scaleable backend, we're going to illustrate a couple of them highlighting the pros and cons of each solution:

Clustering:

"A computer cluster is a group of tightly coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. The components of a cluster are commonly, but not always, connected to each other through fast local area networks. Clusters are usually deployed to improve performance and/or availability over that provided by a single computer, while typically being much more cost-effective than single computers of comparable speed or availability." (from Wikipedia)

SmartFoxServer offers different options for running your applications on multiple servers:

• You can run multiple server instances in parallel by splitting your virtual world into regions and have each of them handled by one or more machines. With this technique you can create large MMOs and gaming systems at a fraction of the cost of a "traditional" clustered system.
  
  The only drawback is that each server runs isolated from the others so if you need to share data across multiple instances you will have to use a central database which could become a bottleneck.


• You can integrate Terracotta with you server side extensions to create a highly available and scalable service with no single-point-of-failure. In May 2008 we officially released an in-depth white-paper that illustrates how to create a SmartFoxServer cluster for large MMOs and on-line game systems.
  
  Terracotta is currently the leading technology for clustering the Java Virtual Machine, it operates behind the scenes at byte-code level, providing unprecedented scalability and fail-over resilience.
  
  From an economic standpoint the integration of Terracotta with SmartFoxServer makes it possible to create large distributed gaming systems at a fraction of the cost of traditional solutions (based on caches and expensive databases).

Parallel instances:

An alternative to increase the scaleability of the socket server is to run multiple parallel instances. By splitting the virtual world into regions we can make each instance handle one portion of the world and share the common data in the clustered database server, which will be available to all regions.

Using this technique, the load balancing is delegated to the client application: the client will either automatically choose the right server based on the current traffic, or present a list of servers to the user allowing him to choose the destination.

( You can read more about performance and scalability in this f.a.q. )

The following diagram illustrates the final server architecture of our fictitious VirtuaPark project:

• Web Server: since most of the server side calls will be directed to the SmartFoxServer extension, including user registration, password recovery etc..., we will probably be able to handle all the web traffic with a single web server machine. Additionally, in order to avoid a single point of failure, we may want to keep a spare mirror machine in case the web server goes down.
  
  
• SmartFoxServer: each instance will run on a high-spec dual-core, dual-CPU machine capable of handling at least 8 - 14.000 users (this includes chats, instant messenger, multiplayer games, etc...) for a total of 40.000 or more concurrent users.
  
  
• Database Server: the database will process the requests coming from all region instances. For this reason we decided to use a clustered solution, in order to guarantee the best availability of the service.

» Securing the Virtual World

We've seen that MMOGs are pretty complex applications based on various server technologies, and for each of them we could write an entire book that delves into specific security concerns. Since this article is centered around the SmartFoxServer technology we will take a look at some common sense techniques and best practices to reduce the amount of hacking to the minimum possible.

• Basic security tools: SmartFoxServer comes with a basic set of security tools that can be configured to avoid typical hacking attempts like connection and message flooding. To avoid these problems we provide anti-flood filters and an IP filter that prevents too many connections from a single IP address. Additionally the server is extremely paranoid with connected sockets that don't perform a login request. Such connections can't interact with the server and they are shut down after a configurable amount of time.
  
  
• Server public requests: SmartFoxServer exposes a number of public commands that any client can invoke on the server, provided that they successfully logged in. Among these requests we have: login request, join request, public and private messages etc... All these public commands are heavily validated by the server to avoid malicious requests and they can be inhibited through the configuration file, to avoid unwanted requests. You can learn more about disabling public requests here.
  
  
• Server side logic: it is vital that all the application logic is kept on the server side, as the Flash client can be easily reverse engineered and modified to perform malicious calls. Additionally you should be paranoid with every client request and heavily validate them, especially in prize-based games, transactions that involve money informations and similar ...
  
  
• Money / Prize transactions: if you run prize-based games (whether it's money, goods, services etc...) you will certainly attract the interest of hackers. In order to achieve the best security you should run your prize transactions between SmartFoxServer and an external SSL-enabled web server. This way the communications happens between the two servers, hidden from indiscrete eyes and through a highly secure protocol.
  
  
• Client side hardening: since SWF files are easily reverse engineered with a decompiler, it is crucial to make the life of the hacker-wannabe as hard as possible. There probably isn't a perfect solution for completely securing the client application, but you can achieve good-enough results by combining various techniques:
  
  
  1. Make your code modular and make it load different external pieces. Also try to obfuscate the file-name strings inside your swf files.
     
     
  2. If you are loading external XML data containing sensitive informations about other application resources, encrypt it.
     
     
  3. Use a code obfuscator: the decompiled source code will become extremely hard to read and understand.
     
     
  4. Use the cross-domain policy file to stop unwanted domains. If someone steals your client application and tries to upload an hacked version on another website, the connection to your servers will be refused.
     
     
  5. With the latest ActionScript 3.0 you can transfer entire swf files as byte arrays through the socket. By doing so you will skip the browser cache and make it very hard to capture.
     
     

» Development tips and tools

Now that we have analyzed the client/server architecture, scalability and security concerns, we should finally be able to put our hands on the keyboard and jump right into coding...

Even if the temptation is strong we should hold on for a moment and discuss a few more aspects of the development that we haven't mentioned so far. VirtuaPark will probably be a great success if we clearly have in mind what we want to achieve, and if we have set a number of realistic goals in terms of money, time and resource investments.

The following is a list of important aspects that should be taken into account before starting to develop our virtual world:

• Planning: as obvious as it may sound, a project like this requires careful planning. It is really advisable to write down one ore more documents that contains all the aspects of the application, the architecture overview, a list of features, the estimated resources needed and a small business plan.
  The document typically requires a number of reviews and, as you get more into the details, you will probably realize that the it won't be possible to pack all the features in the first release. You will probably need to split the project in multiple development phases so that you can set a realistic schedule for the initial release.
  Additionally, starting (relatively) small lowers the risks of failures because you can receive user feedback at an earlier stage and fix what is not working before it's too late.
  
  
• Prototyping: creating quick-and-dirty prototypes is usually a great way to verify if an idea will work as expected. When facing new challenges like those involved in the creation of an MMO, this is the best approach to avoid rewriting large pieces of code that were supposed to "just work".
  SmartFoxServer allows to quickly prototype server side code with highly productive scripting languages such as ActionScript and Python which can significantly cut down the development/prototyping times.
  
  
• Documentation: keeping documents that describe the application in all its aspects is essential to avoid forgetting ideas that you have discussed and never put into writing. Additionally it makes it easier to instruct new members of the team or to explain the project to people outside the company.
  The same goes for both client side and server side code. Maintaining the habit of adding clear comments to the source code will literally save hours of staring at the screen with no clue about where to start, and avoid tedious debugging sessions.
  
  
• Versioning: this is an essential tool when coding in a team. By keeping your code versioned you reduce the chances of loosing code, you get a central repository for the project files and you can let multiple developers work on the same files simultaneously. When everyone is finished coding they just need to submit their changes to the central repository. Modifications made by other team members will be merged automatically, and in case the same part of code was modified by another developer, you will be asked to resolve the conflicting code sections. If at any time a new piece of code breaks the application you can quickly roll back to the previous version without wasting precious time.
  There are many commercial and free versioning tools available, we recommend starting from SVN, which is free and open-source.
  
  
• Logging: debugging multi-user applications can sometimes be an horrible nightmare. In order to avoid long and frustrating debugging sessions, it's highly adivsable to log a lot of informations on both client and server sides. This will allow you to easily check what's going on on both ends of the application while testing.
  SmartFoxServer utilizes the JDK Logging API to output informations to the OS console and log files. The API are also accessible through the extensions, using any of the available languages (java, actionscript, python)
  
  
• Beta phase: a limited public beta phase is essential to do the final testings. Once the application has been sufficiently tested internally, it is time to publish it live and let a selected group of users play with it and report issues. This can be done in various ways: by recruiting professional beta testers, by inviting selected users / gamers in signing up for beta access, by letting anyone register for a beta account without restrictions.
  The important part of this phase is to receive as much feedback as possible from players, in order to seek and squash the final bugs. This also allows you to see how many resources are being used in a real-life environment and foresee what will happen when the application will reach a larger audience.

» SmartFoxServer based MMOGs

SmartFoxServer powers some of the most creative and funny Flash-based communities and MMOGs around the web. You can check our Showcase Section and learn more about each project.

We'd like to mention Disney's Club Penguin (probably one the largest MMO ever created with Flash technology) which has become very popular in many countries all over the world. The community offers a highly interactive environment with hundreds of customizable items and furniture, engaging games, events and quests.

Galaxseeds is another great examples of an MMO with a unique science-fiction theme, games and tons of features.

Build-A-Bearvill is an great children oriented MMO with educational quests and adventures.

Gogofrog is an interesting experiment in social networking and MMOs that uses a real 3D engine optimized for Flash. The community allows users to create and customize their own "unlimited" spaces and share photos, music, documents...

YoVille is a colorful virtual world which integrates with Facebook and has more 150,000 daily active users.

» Conclusions

This article just scratches the surface of one of the most exciting and complex type of multimedia software development, we hope it will serve as an appetizer for further investigation and learning. As usual feedback is always welcome. You can leave your comments in our SmartFoxServer forum.

« Back to part 1 of the article