all users are suddenly disconnected

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

Moori_sx
Posts: 3
Joined: 28 May 2018, 18:40

all users are suddenly disconnected

Postby Moori_sx » 28 May 2018, 20:29

Hello,

I'm running smartfox2x with elastic license and from time to time, not regularly, all sessions are closed and players disconected and all conections are refused for about 5 minutes and then everything goes back to normal.
We're way below our ccu limit and the VM hosting the server doesn't have any peak in memory usage, network traffic or cpu usage.

This is a log for one of those occurancies:
28 May 2018 | 18:24:17,920 | INFO | SFSWorker:Ext:15 | Extensions | | {WarServer}: [ANTI-CHEAT][GameState.OnAttack] CASSIO1914 tentou fazer um ataque a partir de um território que possui apenas 1 exércitos.
28 May 2018 | 18:29:13,695 | INFO | Scheduler1-thread-1 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 54573, Type: DEFAULT, Logged: Yes, IP: 189.115.104.189:60580 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler1-thread-1 | bitswarm.sessions.DefaultSessionManager | | Terminated idle logged-in session: { Id: 54573, Type: DEFAULT, Logged: Yes, IP: 189.115.104.189:60580 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler1-thread-1 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 55338, Type: DEFAULT, Logged: No, IP: 191.180.112.69:63420 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler-1395866287 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 55220, Type: WEBSOCKET, Logged: Yes, IP: 179.235.204.228:59608 }
28 May 2018 | 18:29:13,695 | INFO | qtp65235396-153 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 54877, Type: WEBSOCKET, Logged: Yes, IP: 179.95.160.199:62790 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler-1395866287 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 55221, Type: WEBSOCKET, Logged: Yes, IP: 179.235.204.228:59610 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler-1395866287 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 54847, Type: WEBSOCKET, Logged: Yes, IP: 200.195.176.196:53390 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler-1395866287 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 55047, Type: WEBSOCKET, Logged: Yes, IP: 177.53.100.214:58765 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler-1395866287 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 47707, Type: WEBSOCKET, Logged: Yes, IP: 201.17.211.44:16743 }
28 May 2018 | 18:29:13,695 | INFO | Scheduler-1395866287 | bitswarm.sessions.DefaultSessionManager | | Session removed: { Id: 50581, Type: WEBSOCKET, Logged: Yes, IP: 201.6.234.62:20934 }

the first message is expected in our extension and indicates normal operation, the next hundreds are sessions been removed followed by hundreds of user disconnected logs.

Any suggestions on how to solve this issue?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: all users are suddenly disconnected

Postby Lapo » 29 May 2018, 08:17

Hi,
it's difficult to say what might be going on. The server never disconnects anyone, unless a user has been idle for too long.
When things like these happen they are usually caused by external factors.

Is the server running in a dedicated hosting environment?

It would be very interesting to have a couple of screenshots of the server state during those 5 minutes where connections are not possible, as you described. Especially the states of the queues. (from AdminTool > Dashboard > Message queues)

To further investigate we would need to see the complete logs around the time of this occurrence.
You could zip them and send them to our support@... email box with a reference to this thread.

Also what version of SFS2X are you running?

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
Moori_sx
Posts: 3
Joined: 28 May 2018, 18:40

Re: all users are suddenly disconnected

Postby Moori_sx » 30 May 2018, 14:55

Hello,

I didn't had access to the admin during those 5 minutes because the admin users were also disconnected and unable to connect during that down time.

But I found out that the VM had actually run out of memory and I belive that was the reason for all the sudden disconnects.

I've upgraded the VM and am, now, looking to what seems to be a very slow memory leak.
I was expecting to see a consistent memory/rooms ratio (checking the memory on the valley, after garbage collection). But it creeps up and in 24h the process is taking 4.3GB.

We're managing any static reference, and all other allocations should be freed after the room is removed, right?
I've tried disabling each zone one by one, closing all rooms, to see if the memory's baseline drops, but it didn't. Is that normal?
I've also found a some rooms tagged to remove when empty with no players. Is that normal?

Also, very important, It's not uncommon to the server switch the license back to community for no reason. It happened just now, and using elastic license and have enought credits til 26 Jun 2018. The "Test License" button show a valid 2000ccu license, but I have to restart the server to activate it again.

I'm sorry to mix many topic on the same post, but the forum seems to be the only available contact and I it's been a bumpy release.

Thank you.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: all users are suddenly disconnected

Postby Lapo » 30 May 2018, 15:19

Hi,
Moori_sx wrote:But I found out that the VM had actually run out of memory and I belive that was the reason for all the sudden disconnects.

Ok, this makes sense. If there's an OM, connections will be shutdown abruptly.

We're managing any static reference, and all other allocations should be freed after the room is removed, right?

Normally yes, but there're many ways to introduce memory leaks.
I could suggest to look into leaked database connections, or leaked Schedulers and Scheduler Tasks, but there could be other things in your code that could do that as well.

There are multiple ways to investigate this further:

1) Try to reproduce it locally on a small scale with maybe a few hundreds automated clients and monitoring with tools like VisualVM
2) Plugging VisualVM (or equivalent) directly into the production SFS2X instance to get a better sense of which objects are being leaked.

Here are a couple of useful resources for testing locally and using VisualVM:
http://docs2x.smartfoxserver.com/Advanc ... ive-server
https://smartfoxserver.com/blog/buildin ... test-tool/

I've tried disabling each zone one by one, closing all rooms, to see if the memory's baseline drops, but it didn't. Is that normal?

The GC is non-deterministic, so you don't necessarily see a change in memory usage when you remove a number of Rooms.
I've also found a some rooms tagged to remove when empty with no players. Is that normal?

Game Rooms by default are destroyed as soon as the last player leaves. I am not sure what you mean when you say "Tagged to remove". There is no such mechanism in SFS2X. (i.e. tagging for later removal)

Also, very important, It's not uncommon to the server switch the license back to community for no reason. It happened just now, and using elastic license and have enought credits til 26 Jun 2018. The "Test License" button show a valid 2000ccu license, but I have to restart the server to activate it again.

This is a different issue and sounds strange.
For the license problem I would recommend to get in touch directly with support@... with the details of your license and an extract of your logs when the license reset happens (there's always a trace in the logs when that happens)

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
Moori_sx
Posts: 3
Joined: 28 May 2018, 18:40

Re: all users are suddenly disconnected

Postby Moori_sx » 04 Jun 2018, 20:15

Hello,

Game Rooms by default are destroyed as soon as the last player leaves. I am not sure what you mean when you say "Tagged to remove". There is no such mechanism in SFS2X. (i.e. tagging for later removal)

What I meant was that the rooms with auto-remove mode:WHEN_EMPTY are not been removed when the user count reaches zero. Well, most of the times they do, but not always, and that leaves a few abandoned rooms that has to be removed manually.
http://prntscr.com/jqu06u
This is the creation code:

Code: Select all

public Room createNewRoom(boolean isPrivate) throws SFSCreateRoomException {
      CreateRoomSettings roomSettings = new CreateRoomSettings();
      // roomSettings.setGroupId(AvailableGroup);
      roomSettings.setDynamic(true);
      roomSettings.setAutoRemoveMode(SFSRoomRemoveMode.WHEN_EMPTY);
      roomSettings.setGame(false);
      roomSettings.setMaxUsers(6);
      roomSettings.setMaxSpectators(0);
      roomSettings.setName((isPrivate ? "P_" : "R_") + UUID.randomUUID().toString().substring(0, 8));
      roomSettings.setExtension(new RoomExtensionSettings("WarServer", "extension.WarRoomExt"));
      return getApi().createRoom(getParentZone(), roomSettings, null, false, null, false, false);
   }


Thank you.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: all users are suddenly disconnected

Postby Lapo » 05 Jun 2018, 08:56

Hi,
it's difficult to say what might be going on with the hanging Rooms.
It's possible a runtime error caused this, we should inspect the logs to have more clues. Also it might be linked to the other memory issues you've reported.

If you've ongoing memory leak in the Room Extension, all kinds of strange stuff can happen...
Were you able to find what was causing the OutOfMemory issues?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
rewb0rn
Posts: 177
Joined: 26 Jan 2009, 12:54
Location: Berlin, Germany

Re: all users are suddenly disconnected

Postby rewb0rn » 18 Jun 2018, 22:16

Hello,

I wonder if we are also seeing an out of memory problem on our machine.. Are there any logs by smartfox when an our of memory error happens? If not, how would we detect that the memory limit is reached on a production Ubuntu system?

Thanks in advance
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: all users are suddenly disconnected

Postby Lapo » 19 Jun 2018, 07:20

Hi,
yes most of the times (I'd say 90%+) you will find a OutOfMemoryError in the logs. It's unlikely that a JVM crashes immediately upon OOM without traces of it.

Memory problems can be detected by keeping the Dashboard open for several hours and checking the trends of memory growth.
Here's a detailed guide on it: http://docs2x.smartfoxserver.com/Advanc ... ive-server

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 48 guests