High cpu usage due to Thread.sleep

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 21 Oct 2018, 16:50

Hello
we recently moved to SFS2 after using sfs pro for almost 10 years :D, the last to bottle neck our stack was the CPU, our game isn't very CPU demanding, it's turn based and not so much math to do

we ran the servers, and the CPU load is increasing dramatically with the users joining, we have only 1.6k users connected to the server and the CPU load is almost 100% now
the game is a card game where only 4 users are in a room (might have spectators), so we have around 350~ rooms in the server

the CPU load doesn't make sense at all, so I tried to use profiler tools (used yourkit), and I found that 80% of usage is due to Thread.sleep method
we are not calling this from our code at all, so I need to get some insights from you guys to figure out where is this coming from

Here is a picture from the profiler
Image
please not this is note the full list, all the way down there are more Thread.sleep taking 20% each, the methods from our code are barely responsible for 13% for the overall usage

and here is more details about the callers
Image

please help us with is, when does SFS use thread.sleep ? is it a miss use of the Scheduler ? is the Scheduler blocking ?
we use the system scheduler in rooms, we don't create new ones, when AI plays for example we delay its action for x seconds, or to give players timer to play, once finished the player is forced to take action and so on

is this a miss use? or I'm totally off ? could it be SQL querys ? we also use the SFS DB class, is this blocking? although we used to do the same thing in the old servers
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High cpu usage due to Thread.sleep

Postby Lapo » 22 Oct 2018, 08:07

Hi,
I think you might be interpreting those numbers the wrong way, but I am also not familiar with Yourkit profiler and I am not sure what I am looking at.

I would rather like to see a screenshot of the SFS Admin Dashboard where we can see the process CPU usage vs system CPU usage and a breakdown of each thread's CPU usage as well.

Thanks.
Lapo
--
gotoAndPlay()
...addicted to flash games
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 22 Oct 2018, 08:55

Here is an image of our smart admin tool
Image

The process is consuming almost everything, there isn't much going on the server other than smartfox
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High cpu usage due to Thread.sleep

Postby Lapo » 22 Oct 2018, 09:50

Thanks,
this gives us a better picture of the state of the server.

If you take a look at the "Active Threads" window you will see that the threads using more CPU time are the "pool-1-thread-*" and the 4 extension workers. In other words those 8 threads make up for the ~75% of the CPU usage you're seeing in the main graph.

Since pool-1-thread-* refers to the global Task Scheduler I think what you're seeing is the result of your Extension code + Task code running in the server.

Is your server side code written in Java or Javascript?
What are the specs of your machine? (CPU/RAM etc...) Also what OS does it run?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 22 Oct 2018, 10:02

The CPU is 4 Cores (Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz)
8GB RAM
and we are runnig centos 7

and yes we are using java for extension code
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 22 Oct 2018, 10:05

Probably this information is important too
I'm getting the system timer and use it inside the game
everytime a user takes an action the timer starts (15 secs for example)
if user didn't play the AI takes over
everytime an AI plays I schulde half a second delay for the AIs

and this is happening in every room, so you can say we have a lot of scheduled small tasks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High cpu usage due to Thread.sleep

Postby Lapo » 22 Oct 2018, 10:33

Hi,
and the machine is a physical one, right? Not virtualized?

In any case it looks like the AI might be part of the "problem", since the threads using the most CPU time are those running in the Scheduler.

Maybe there are optimizations that be can be applied to that part of the code? Or maybe it's just the nature of the game AI which requires a certain amount of CPU time.

From a "health" standpoint, the screenshot you sent don't show anything anomalous in the server, other than showing a pretty busy SmartFox.

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 22 Oct 2018, 10:39

I doubt the AI code is the issue, it's not real AI that involves MinMax or anything heavy
it's a little bit long but not more than few conditional cases and reading few arrays

as far as I know it's a virtual machine
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High cpu usage due to Thread.sleep

Postby Lapo » 22 Oct 2018, 12:48

Ok,
so if it's a virtual machine and the code you're running is not that demanding it might be an issue with lack of resources, plain and simple.

The quad core Xeon is pretty good and natively it would be expected to run many more concurrent players than those you're handling now. However if it's a virtualized environment we don't know how many CPU resources/cores are really available at any one time.

In any case it's indicative that the threads that are most expensive are those running your code. This means that the traffic is not putting any pressure on the server core (otherwise its threads would jump up in thread list) and Extensions + Schedulers is what uses more CPU cycles.

You mentioned reading a few arrays. Is it possible these arrays are quite big (10K+ items) and are being read tens of times per second?

Also, I think with your profiler you should be able to actually dig into the specific method calls that are using the most resources.
Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 22 Oct 2018, 16:03

No the arrays don't exceed 10 at most
Thanks for the information, this is helpful, I'll post back if I find anything else or if I solved the issue
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 25 Oct 2018, 19:22

so After a lot of debugging and profiling, turns out Thread.sleep is irrelevant, it's showing this result as "Wall Time", long story short, I excluded the method analysis from the profiler and I found few hot spots to optimize and I lowered the CPU usage a little bit, still not good enough though

the highest usage now is due to db.getConnection, I tried to increase the max allowed connection and max idle connections to 150, 60 to check if we have a problem here, the server looked stable at first and CPU load came down to 20%, but now we are back to 50% with the same CCU just later that day

I'm not even sure I should increase the connections to this crazy number, if DB connections are causing the issue shouldn't it be like just a long wait time? is this number even acceptable ?

we do have a LOT of queries, every round is logged, and our login is also big, user inventory and lot of other stuff, I'd say our DB is busy, but I have the settings for the DB to GROW anyways

is there anything you can tell me about this?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High cpu usage due to Thread.sleep

Postby Lapo » 26 Oct 2018, 07:19

I find it a bit strange that most of the CPU time is spent in that method.
Can you show us a profiler's screenshot with the details?

In any case you can monitor the number of DB connections used via the AdminTool's Zone Monitor, under the "Runtime Zone Settings" tab, at the bottom it will tell you the number of active connections at any one moment.

You can check that out at peak times to see what's going on.
Lapo

--

gotoAndPlay()

...addicted to flash games
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 26 Oct 2018, 12:05

I din't know about that thanks! but now you mentioned it the numbers are weird
I have 0~2 active db connections and around 4 idle connection while 1.5k CCU users are active

I can guarantee that not a single second would pass without the server making queries, can you explain this?
Also, I noticed when I restart the server the CPU load is only around 20% for the same number of users, after a while the load increases, the memory is fine though, and I'm using one DB object across all rooms

I'll take a fresh profiling session and upload the results

thanks
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 26 Oct 2018, 12:34

This is the hotspots provided by the profiling tool
https://drive.google.com/file/d/1GZTCuN ... sp=sharing
Hassan Khallouf
Posts: 41
Joined: 06 Jul 2017, 04:35

Re: High cpu usage due to Thread.sleep

Postby Hassan Khallouf » 27 Oct 2018, 13:40

Behaviour update:
the load around 1.2k CCU is always around 20%, at peak times we have 2k CCU the load is 70~80%

when users go back to 1.2k CCU the load isn't magically back to normal it takes a while for it to re-stabilize on 20% again

I read in on of your articles that if you are going to schedule hundreds of jobs it's better to use zone level scheduler, I'm still using the system scheduler, could this be the issue?

in our old system (SFS pro) we used to have 1 Thread per room created and managed manually by us directly extending java's Thread

Return to “SFS2X Questions”

Who is online

Users browsing this forum: Justinzeaidew and 48 guests