SmartFoxServer Pro Benchmarks

Document revision 1.0.0 - 2007

The following document presents the results of several benchmark tests done with SmartFoxServer PRO during September 2007. These tests were created in order to measure the server impact on hardware resources, network throughput and overall stability.

Test Machines and Environment

All the benchmarks were conducted using the following hardware/software configuration:

» Server:

» Clients:

Two additional laptops were used for monitoring the server, clients and network traffic.
All computers were connected in the same local network through a 100Mbps switch.
Each test was performed at least 3 times and the results represent an average of the data gathered in each test iteration.

Benchmark #1 -- Chat / Room creation test

» Client requests sequence:

• User logs in the Zone (using a custom login handled by a Java extension)
• User joins a room: a server side extension automatically searches for the first available room.
  If none is available a new chat room is created before the user is joined.
• User sends a chat message (50 chars long) every 5 seconds, this is repeated in an endless loop.

» Results:

Concurrent clients:	5.000
Client connection speed:	10 clients/s
Total rooms:	1000
Network throughput:	~ 55 - 60 Mbit/s
CPU occupied:	5 - 6%
Dropped messages:	none

» Analysis:

The test simulates many concurrent clients creating small chat rooms and sending continuos chat messages. The network traffic is pretty high since each new room created generates updates that are broadcasted to all clients. Also each join request produces more "uCount" messages that are sent to all the connected clients.

The custom logic used for the handling the login and room management, a Java extension, adds almost no overhead to the server performance.
Disabling the extension didn't show noticeable differences.

The server engine handles the stress test very easily, without any slowdowns and with a consistent throughput. At the end of the test all clients are suddenly disconnected, the remaining ghost connections are handled gracefully after the maxUserIdleTime is expired.

Benchmark #2 -- Chat / Room creation test (extended)

» Client request sequence:

• Same as previous test

» Results:

Concurrent clients:	7.500
Client connection speed:	10 clients/s
Total rooms:	1.500
Network throughput:	~ 86 Mb/s
CPU occupied:	18 - 20%
Dropped messages:	~ 12K

» Analysis:

This test extends the previous one by stressing the available network capacity (100Mb/s switch) to the limits. We aimed at connecting 10k users but the available bandwidth was almost full at around 7200-7300 connections.
The server machines used for simulating all the concurrent clients where very busy (cpu at 80-85%) and SmartFoxServer started to drop messages in order to avoid growing the message queues indefinitely.

Additional traffic was also created by idle users that where disconnected for inactivity. In fact when the bandwidth was almost saturated, some of the clients were unable to write data to the socket and exceeded the idle timeout. This generates more updates that need be broadcasted back to all clients.

The server engine handled the stress test very easily, without any slowdowns, with a consistent throughput and protecting itself from out-of-memory errors that could be caused by the network congestion. Messages were dropped only when the outgoing client queue was full (we configured a max. of 120 messages for each client).

Benchmark #3 -- Massive Private Message Test

» Client request sequence:

• User logs in the Zone
• User auto-joins the main Limbo Room, which has a max. capacity of 100.000 users.
  
• User sends a private chat message (20 chars long) to the user whose id is equal to the currentUserId - 1, every 3 seconds.
  This is repeated in an endless loop.

» Results:

Concurrent clients:	20.000
Client connection speed:	10 clients/s
Total rooms:	1
Network throughput:	~ 28 Mbit/s
CPU occupied:	22 - 24%
Dropped messages:	none

» Analysis:

This test highlights the raw performance and throughput of the server engine: the clients are connected in a ring-like pattern where each user sends and receives private messages from the client whose id is equal to currentUserId-1 and currentUserId+1.
It is interesting to note that even 20.000 concurrent connections take less than 1/4th of the CPU resources, leaving tons of spare CPU time for custom extension logic. Additionally even only 256Mb of heap memory in the JVM are more than sufficient to cope with the high load.

The server engine handled the stress test very easily, without any slowdowns and with a consistent throughput. At the end of the test all clients are suddenly disconnected, the remaining ghost connections were handled gracefully after the maxUserIdleTime is expired.

» Additional resources:

• A screenshot of the SFS AdminTool during one of the tests
• The server logs after one of the test iterations

Conclusions:

Paired with the latest Java Runtime 1.6, the SmartFoxServer PRO engine displays outstanding scalability and stability, allowing tens of thousands of simultaneous clients interacting with each other, even on relatively inexpensive hardware.