3.6 Troubleshooting common runtime problems

» Introduction

In this article we are going to discuss several common runtime issues you might encounter while running SmartFoxServer that can be usually solved with a few changes in your configuration, JVM setup or server settings.

» Problems connecting to the server.

Among the common help requests we receive there is the inability to connect or reach the production server once the application goes live for the first tests. There are a couple of "popular" pitfalls that are often overlooked and that can cause a few headaches:

• Server side firewall/nat configuration issues
• Flash Player security restrictions
• Client side firewall or connection problems

» Firewall and Nat:

Once your SmartFoxServer instance is up and running on your server machine at the hosting center you should make sure with the system administrator that there is no firewall blocking TCP connections. If this is the case you should ask the admin to open the TCP ports you are using (default = 9339 and 8080 for the BlueBox) to "the world".

If your server is not connected directly to the internet you should also make sure that the NAT service on the router you are connected to is configured correctly.
The NAT service allows to expose one or more services from your computer to the public internet address. For example a well configured router will send all traffic coming to port 9339 (SFS's default) to the server machine running SmartFoxServer.

In order to determine if something is blocking your server at the "hosting level" you can try telnetting its public address from outside. Telnet is a command-line utility found on most operating systems that allows to establish a TCP connection to a remote host. Follow these steps to test the connection to your server:

• Open the system console:
  • Under Windows use Windows Key + R and then type cmd in the dialogue box that appears
  • Under MacOS X the terminal is found in /Applications/Utilities/Terminal
  • Under Linux... well, every Linux user knows how to launch a terminal!


• Type the following:

  telnet 10.20.30.40 9339



• You should get something like this:

  Lapo$ telnet localhost 9339
  Trying ::1...
  Connected to localhost.
  Escape character is '^]'.
  <cross-domain-policy><allow-access-from domain='*' to-ports='9339' /></cross-domain-policy>

  This is the server accepting the connection and responding with the Flash cross-domain policy.
  If the connection fails you should make sure that no firewall is blocking the port(s) you are using. Additionally you should also check with the admin that the NAT, if any, is properly configured.

» Flash Player security restrictions:

If the server is reachable at its public address but you are still having connection problems, you should make sure that you are not incurring in any Flash player sandbox restrictions. For security reasons the Flash Player does not allow connecting to external domains without setting explicit permissions via a cross domain policy file.

You can learn more about the Flash Player security settings in this white-paper.

» Client side firewall or connection problems:

The last case is when a specific user or group of users are not able to reach the server besides the aforementioned possible issues.
In this case it is highly likely that the clients are under restricted network conditions: for example they run a personal firewall that blocks socket connections to SmartFoxServer, or their connection passes through a proxy server which doesn't allow a point-to-point connection.

In all these cases the SmartFoxServer BlueBox can come to rescue. The BlueBox is an add-on specifically designed to bypass these restrictions and allow users to fully enjoy any SmartFoxServer-based game and MMO.

You can learn all the details about the BlueBox by visiting this section of our website.

» Understanding Dropped Messages.

Among the questions asked in our support board many revolve around the topic of dropped messages. At times you might notice a very high count from the Admin Tool especially for the outgoing dropped messages.

The Incoming Dropped Messages are very simple to explain: requests are never dropped unless their size is greater than the <MaxMsgLen> setting: this allows to put a limit to the size of incoming data and discard any other message with a larger size.

The Outgoing Dropped Messages value keeps a count of the server responses that were not sent to their respective recipients.
Before discussing the possible reasons for dropping a message, let's take a look at the following diagram:

The server associate an outgoing message queue to each connected user in order to store data that cannot be written immediately. In fact the server attempts to deliver each response as quick as possible, but there can be times in which the socket connection is congested and refuses to get more data for a specific client.
In this case the server can only store the remaining bits of data in the queue and wait for the network pipe to become available again for receiving data.

According to the user configuration, SmartFoxServer will keep data in each queue until its capacity is exhausted: at that point messages are dropped. This mechanism allows the server to protect itself from indefinite memory allocation which might eventually end up in a crash of the Java VM.

Here's an example of the server configuration settings:

<ClientMessagQueue>
	<QueueSize>120</QueueSize>
	<MaxAllowedDroppedPackets>3</MaxAllowedDroppedPackets>
</ClientMessagQueue>
	

With these settings the server allocates a queue of 120 messages for each client and allows a tolerance of three dropped messages before disconnecting the client. In general dropping messages is a never a good thing as the user might loose one important packet of data that leaves him out of synch with the rest of the players. For this reason we usually recommend to fine the tune queue size and allow very few dropped packets before disconnecting the client. (Ideally the threshold should be == 1)

» What are the causes of dropped outgoing messages?

• Bad or slow client connection: the user has a slow response time and not enough bandwidth to keep up with the server responses. This causes the socket to become busy very soon and forces the server to keep the data in the queue.


• Too much data being sent too frequently: this is a slight variation on the previous theme. Only this time we can not necessarily blame the client connection but it is probably depending on the server logic sending too large data or too frequent updates. It is important to remember that over the internet every connection experiences a certain amount of latency which usually varies between 50 to 200 milliseconds, but it can also reach several seconds.
  
  It is important to keep in mind these limitations and work around them with specific client and server algorithms to reduce the lag. In general it is not recommended to send more than 10-20 updates per second, unless you're in a local network.
  
  We also suggest to use the string-protocol to reduce the packet size to the minimum and for fast updates (e.g. player position updates in real-time action games)
  
  If you are interested into learning more about network lag we suggest to read this article.


• Lack of bandwidth on the server side: as you reach the capacity of your hosting bandwidth you will experience a general slow down of the network performance. Your players will probably start to complain about the game or application getting slower and you should see a rising number of dropped messages.
  
  You can easily detect this issue by monitoring your bandwidth usage on a daily basis.

» Ghost connections.

Ghost connections are the result of an incomplete socket disconnection which is not acknowledged by the server and can result in a few different problems: re-connecting users be refused because the server thinks that they are still online, the other players in the room will still see the disconnected user online etc...

In reality these pending connections are not fully closed, the TCP stack is still holding them and it's likely that several minutes will pass before the TCP timeout is fired and the connections are finally closed. When this happens the JVM is informed and finally SmartFoxServer can remove the owners of those connections.

The causes behind these "bad disconnections" can be multiple: an abrupt internet disconnection, the network cable being pulled, a crash in the browser or Flash plugin, a disconnection happening from a very slow and problematic connection, etc...

Since this problem is not avoidable SmartFoxServer provides several tools to deal with it:

• <MaxIdleTime> setting: Allows you to specify the maximum amount a client can stay idle (not sending any requests) before being auto-disconnected.
  The server continuously monitors the connected user activity and attempts to kick the idle ones, including ghost-connections.


• <MaxSocketIdleTime> setting: This is similar to the previous one but it controls only those socket connections that haven't performed any login yet. This is useful for getting rid of idle connections coming from bots or any other "automated spiders" that scans the web for attackable services.


• force login option: Since SmartFoxServer PRO 1.6.3 we have introduced an extra "force login" flag in the server side login() method that forces the sudden disconnect of a previous user when the same client is logging in again.


• manual disconnection: If it's a matter of only a few ghost users you can try to disconnect them manually using the SmartFoxServer Admin Tool.