Server CPU Utilization spikes on multiple users disconnecting at the same time

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby sitnik » 17 May 2018, 05:57

Hi ,

I am facing a server CPU spike when huge no. of users get disconnected at the same time, however server is running fine, but still i would like to know the reasons behind it !!

Are there any heavy operations on SFS level which occur at the disconnection time?

Observations:-

I have certain DB operation in our user disconnection handler.

Earlier along with this db operation when i disconencted around 10000 users then the CPU spike firslty goes down to the lowest from 30%, then suddenly spikes to 60 %, within few seconds it came back to normal.

After removing the DB operation from disconnection handler now the spike firslty goes down to the lowest from 30%, then suddenly spikes to 48% , again within few seconds it came back to normal.

Attaching the screenshots of SFS admin dashboard for the above mentioned scenarios.

SFS server version - 2.9.2

Please help me understanding what could be the possible causes of such spike.
Attachments
10000RemovedDirtyDB.png
Without db operations in disconnection handler
(231.54 KiB) Not downloaded yet
10000Init.png
With db operations in disconnection handler
(255.01 KiB) Not downloaded yet
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby Lapo » 17 May 2018, 07:34

Hi,
if you disconnect 10K users at once you're indeed asking the server to do a lot of clean up job. It is expected to see a CPU spike.

Every user disconnect requires cleaning rooms, variables, buddy lists, updating all other clients etc...

Cheers
Lapo
--
gotoAndPlay()
...addicted to flash games
sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Re: Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby sitnik » 17 May 2018, 07:49

Lapo wrote:Hi,
if you disconnect 10K users at once you're indeed asking the server to do a lot of clean up job. It is expected to see a CPU spike.

Every user disconnect requires cleaning rooms, variables, buddy lists, updating all other clients etc...

Cheers



Hi Lapo,

Is there any way i can have a look on what operations, sfs is performing at particular moment of time? which you have stated above !!
sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Re: Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby sitnik » 17 May 2018, 11:46

Lapo wrote:Hi,
if you disconnect 10K users at once you're indeed asking the server to do a lot of clean up job. It is expected to see a CPU spike.

Every user disconnect requires cleaning rooms, variables, buddy lists, updating all other clients etc...

Cheers



Lapo,

Can you elaborate a little bit about the above mentioned operations, how costly are they, how much do they increase CPU utilization or memory !!
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby Lapo » 17 May 2018, 14:54

I don't think I can do that. :)
CPUs come in all sizes, cores, native threads, clock speeds and such. It is not possible to provide metrics as to how X amount of disconnections can impact the CPU with all these variables.

Also I think you're worrying too much about something that is never been an issue.
1) a scenario where 10K connections all go down at once is extremely unlikely
2) even with that the server (as you mentioned) keeps working as expected

I've already mentioned the kind of work the server has to do when a client disconnects. Clearing internal data structures, dereferencing all "pointers" to the User, destroying variables linked to the User and updating all the necessary clients. This last operating in particular is likely the one causing more work because many updates must be created, enqueued for delivery etc...

It's just necessary work that needs to be done.
Lapo

--

gotoAndPlay()

...addicted to flash games
sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Re: Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby sitnik » 18 May 2018, 05:57

Lapo wrote:I don't think I can do that. :)
CPUs come in all sizes, cores, native threads, clock speeds and such. It is not possible to provide metrics as to how X amount of disconnections can impact the CPU with all these variables.

Also I think you're worrying too much about something that is never been an issue.
1) a scenario where 10K connections all go down at once is extremely unlikely
2) even with that the server (as you mentioned) keeps working as expected

I've already mentioned the kind of work the server has to do when a client disconnects. Clearing internal data structures, dereferencing all "pointers" to the User, destroying variables linked to the User and updating all the necessary clients. This last operating in particular is likely the one causing more work because many updates must be created, enqueued for delivery etc...

It's just necessary work that needs to be done.


"1) a scenario where 10K connections all go down at once is extremely unlikely"
Actually Lapo, we have faced this issue recently when the ISP service went down.


Also please let me know the heavy operation that you mentioned "updating all the necessary clients. This last operating in particular is likely the one causing more work because many updates must be created, enqueued for delivery" , i assume the disconnection of a single user is notified to all the users present in the game room , is there any way i can switch this off, is it optional? Like i dont want the user getting disconnected to be updated to other users present in the room. Is there a way to avoid this? I assume this require a lot of messaging to be done as you stated above.


I found few methods available in Zone api

setUserCountChangeUpdateInterval(int interval)

setClientAllowedToOverridRoomEvents(boolean flag)

Can i use them to avoid this or may be to reduce the CPU load? What are these methods supposed to do? Also what will be their default values in zone if i have not configured them in code.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server CPU Utilization spikes on multiple users disconnecting at the same time

Postby Lapo » 18 May 2018, 07:21

sitnik wrote:
"1) a scenario where 10K connections all go down at once is extremely unlikely"
Actually Lapo, we have faced this issue recently when the ISP service went down.

Sure, but it's not common right? It's not going to happen every day. If it was you should have to change ISP because no one wants a dedicated hosting to die every day :)

i assume the disconnection of a single user is notified to all the users present in the game room , is there any way i can switch this off, is it optional?

You can configure each Room not to send any USER_ENTER, USER_EXIT events, but this way people will not see each other.


Like i dont want the user getting disconnected to be updated to other users present in the room.

If users in a Room are updated when new players enter the Room it's necessary to also update them some goes away. Otherwise the Room gets de-synchronized, and clients will see players that no longer exit in that Room.

It seems to me that you're exaggerating the problem when in reality disconnected users create a minimum amount of work for the server which is negligible and doesn't impact the performance, under normal conditions.

I assume this require a lot of messaging to be done as you stated above.

It's the necessary amount of messages to keep everyone in synch.

setUserCountChangeUpdateInterval(int interval)

setClientAllowedToOverridRoomEvents(boolean flag)

Can i use them to avoid this or may be to reduce the CPU load? What are these methods supposed to do? Also what will be their default values in zone if i have not configured them in code.
i assume the disconnection of a single user is notified to all the users present in the game room , is there any way i can switch this off, is it optional?

Those parameters are available in the configuration of the server, from Admin Tool.
They don't save a little bit of CPU but mostly they save bandwidth.

The UserCount update sends the Room user count updates more or less slowly, according to your parameters.
The other option allows you to change the type of events that you want to send from a specific Room, including the updates for users entering and leaving the Room.

For more you can read here:
http://docs2x.smartfoxserver.com/Develo ... chitecture

under the "Room fine tuning settings" section.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 44 guests