Our server freezes - So many queued threads

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

moonlife
Posts: 45
Joined: 26 Sep 2016, 08:22

Our server freezes - So many queued threads

Postby moonlife » 20 Jun 2022, 12:42

Hi,

We are experiencing a serious problem. Sometimes our game freezes for 10-20 seconds, sometimes for minutes, and sometimes SmartFox Server completely freezes and we need to restart our server. This issue started to happen like a week ago, so we don't think the reason is our server side extension, it was working fine till then. Also our concurrent user count didn't increase that much.

We are suspecting someone sends so many extension requests to our server, but we are not sure why is this happening. Is it possible to detect someone sending so many extension requests?

We are having a lot of threads like this in our thread dump when this issue occurs:

Code: Select all

"SocketWriter::TCP-2" Id=50 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@44984796
   at sun.misc.Unsafe.park(Native Method)
   -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@44984796
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at com.smartfoxserver.bitswarm.core.SocketWriterV3$TCPWriteRunner.run(SocketWriterV3.java:818)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

For example, we had 44 threads like this in our thread dump, and VisualVM showed an error message saying there are deadlocks in our threads. But when we checked our thread dump, we didn't see anything odd except a lot of these AbstractQueuedSynchronizer threads.

We have a lot of game rooms with 5 players each. In our Java extension which players send request when they make a move (let's call it move.java), we are using synchronized(room). We are suspecting that's the reason of so many AbstractQueuedSynchronizer threads, but we still couldn't figure out why a deadlock happens.

Could increasing extension thread count help us on this issue (we know you don't encourage it)? We don't want to get rid of synchronized(room) here because we want to make sure every player makes their move one by one, in order.

Is there anything else to figure out the reason of these delays/freezes on our SmartFox Server?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Our server freezes - So many queued threads

Postby Lapo » 20 Jun 2022, 13:12

Hi,
the stack trace you have posted does not raise any concern. It is perfectly normal for the SocketWriter threads to be in that state.

The slow down/freezes you have described suggest that a message queue might too busy, causing other requests to pile up and response to come very late to each clients.

I would recommend taking a full thread dump of the JVM when the problem appears and also monitoring the server via the Admin Tool's Dashboard. Here you find two sections that show the overall state of the server and the state of its queues.

Let us know what you find.
You can send us screenshots by posting them here or via email at our support@... email box (adding a reference to this forums post)

Cheers
Lapo
--
gotoAndPlay()
...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 58 guests