SOS! 911! Server stops responding many times a day

[DFTGames]

Hi,

we have published Bionauts and now we're in deep water :/ It's a free to play so you can easily try it (needs a good tablet to run!):

Android:https://play.google.com/store/apps/details?id=com.hawkindustriesllc.bionauts

iOS: https://itunes.apple.com/us/app/bionauts/id523265043?ls=1&mt=8

During the Beta we had zero problems. Now we are live and... the server just stops working many times a day . I had to add a cron job to restart it every hour, but really this can't be the solution. I need some pointer to solve the problem.

BEst,
Pino

[rjgtav]

Hello,

What server version are you running? And on what OS?
Does the server throw any error when it shuts down? Please check the logs located on the SFS2X/logs folder
If you check on the AdminTool, is the server working hard, like big CPU usage and/or high memory usage?
And does the server really shut down or simply start hanging during the login phase? If this is the case, please make sure that you return every DB connection to the pool after using it.
Maybe you're running out of memory? Please make sure that you clear everything when, for example, a game ends (in case you're using a Room-Level extension), or clear every room reference when it is being removed (in case you store the list of rooms in a custom list).

Please provide more information so we are able to help you,
Thanks

[DFTGames]

Hi rjgtav,

we have already checked all those potential issues during the Beta. We run a Debian dedicated server, 2 4 core CPU with 16GB RAM. THere are no errors nor warnings and the memory (and queues) are never full, actually they are never even near 30% of load, and the bandwidth is not even at 2% (we have 1Gb always on).

This is driving me crazy

[rjgtav]

Do you have any service or app which is monitoring all the apps running on the server and automatically shutting down the ones that are using significant system resources?
Have you tried increasing the server's max memory to see if it takes longer to stop responding?
And can you confirm that the server does really shutdown?

[DFTGames]

Hi,

yes, the server status is monitored via a monitoring card,so it's HW protected and it reboots in case of any service eating up resources while sending me a text message. I tested this feature making a looping program and it works jkust fine, but there are no services eating up anything... this server is dedicated to this only game, so there is nothing else on that machine that can cause troubles.

SFS is configured to use a minimum of 512MB and a maximum of 4096MB, but it never goes over 160MB of usage. And yes, the server it's really restarting, so say the logs and so says the "Server Uptime" in SFS Admin dashboard.

[Lapo]

During the Beta we had zero problems. Now we are live and... the server just stops working many times a day

Please can you describe what happens when it stops working?
Do you use a database along with SFS?
Does the AdminTool work when the problem shows up?

Next time it happens please grab a full thread dump: viewtopic.php?f=18&t=14458
Thanks

[DFTGames]

Hi Lapo,

yes, we have a DB and that one is responsive as normal. I will create a thread dump next time, but I really hope that it'll not be a next time :/ I'm going to set the restart every 6 hours to see if that incurs in the problem.

[Lapo]

If you are using a database and you are grabbing connections manually (via the getConnection() method) you must pay attention to handle DB exceptions correctly. It happens quite often and the result is an unresponsive server because all Extension threads end up blocked waiting for new connections from the DB pool.

Check the documentation here:
http://docs2x.smartfoxserver.com/Develo ... ustomLogin
under the "Handling Exceptions" section.