Random infinite loop in java.util.zip.Inflater.inflateBytes

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

User avatar
arodrigo
Posts: 24
Joined: 22 Jul 2011, 10:03
Location: Spain
Contact:

Random infinite loop in java.util.zip.Inflater.inflateBytes

Postby arodrigo » 27 Oct 2013, 09:04

Hi,

Since the last version of SFS 2.7.0 sometimes one thread start to consume the 100% of his CPU. At this time the server, drops to users and avoid the connections to the mobile clients but not to the flash clients.

The thread at 100% is:

Code: Select all

"SocketReader" prio=10 tid=0x00007fd6b1ba7000 nid=0x67ce runnable [0x00007fd5952d1000]
   java.lang.Thread.State: RUNNABLE
        at java.util.zip.Inflater.inflateBytes(Native Method)
        at java.util.zip.Inflater.inflate(Unknown Source)
        - locked <0x000000074751b220> (a java.util.zip.ZStreamRef)
        at java.util.zip.Inflater.inflate(Unknown Source)
        at com.smartfoxserver.v2.protocol.binary.DefaultPacketCompressor.uncompress(DefaultPacketCompressor.java:92)
        at com.smartfoxserver.v2.protocol.binary.BinaryIoHandler.handlePacketData(BinaryIoHandler.java:468)
        at com.smartfoxserver.v2.protocol.binary.BinaryIoHandler.handleRead(BinaryIoHandler.java:231)
        at com.smartfoxserver.v2.protocol.SFSIoHandler.onDataRead(SFSIoHandler.java:131)
        at com.smartfoxserver.bitswarm.core.SocketReader.readTcpData(SocketReader.java:338)
        at com.smartfoxserver.bitswarm.core.SocketReader.readIncomingSocketData(SocketReader.java:222)
        at com.smartfoxserver.bitswarm.core.SocketReader.run(SocketReader.java:173)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)


This month two of our servers had this problem.

Let me know if I could provide you more information in order to resolve the problem.

Regards
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby Lapo » 27 Oct 2013, 09:21

Hello,
I don't think 2.7.0 is any different from 2.6 or 2.5 in that department (protocol compression)

Why did you write "infinite loop" in the title? What makes you think there's a loop? From what I can see there doesn't seem to be one. It just seems that one thread is struggling (maybe more than one).

Anyways, we will need much more details to be able to help.
Do these error appear at peak traffic? What is the CCU when this happens? Does it go away on it's own?
What's the overall CPU usage of the server in those occasions?
We will also need a screenshot of the Dashboard main view and Queue's view.
The full thread dump will also help.

Does your application use/sends very large packets from client side? (e.g. > 1-2MBytes)

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
User avatar
arodrigo
Posts: 24
Joined: 22 Jul 2011, 10:03
Location: Spain
Contact:

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby arodrigo » 28 Oct 2013, 09:42

Hi Lapo,

The infinite is only a fast stupid assumption (sorry), the thread remains in this state infinite time .

The CCU the last 3 times was between 800 and 2500.
The CPU in the lowest traffic server (800 - 1000 CCU) was at 5-10% when the thread has failed and in the others servers (2500-5000 CCU) the CPU was at 10 - 30%.
I can't provide you a screenshot of this moment because at this moment we cannot establish connection with the admin.
Our application do not sent very large packets in any direction.

In the following links you can see a thread dump and the graphics of CPU and CCU at the moment of the fail. https://dl.dropboxusercontent.com/u/462346/data.zip

When we take the last thread dump the only process at 100% of CPU was the 0x67ce.

Thank you for the help.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby Lapo » 28 Oct 2013, 10:00

You have a huge amount of threads going on in the system, it's quite difficult to understand what might be going on.
From the graphs you have sent it is not clear what the X axis represents? Is it minutes? days?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
arodrigo
Posts: 24
Joined: 22 Jul 2011, 10:03
Location: Spain
Contact:

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby arodrigo » 28 Oct 2013, 10:21

The X are days and the number is at the 8:00 GMT+1. The last time was between the 26 and 27.

Most of the threads are from ehcache and they are sleeping Ignore them is normal. The systems works fine normally without queue's and with a good ping with 5000 CCU. This error happens randomly with low or high quantity of CCU. And it started with the last version of SFS 2.7.0.

Thanks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby Lapo » 28 Oct 2013, 10:43

When you see this problem in action what do you do? You wait for it to go away? Restart the server?

We don't have other reports of a similar issues and I've personally done lots of stress testing in that department with CCU counts in excess of 30K on a single instance.
I would consider the possibility of some hacking attempts, maybe someone is sending very large compressed packets to slow down the system.

To avoid something like this you can fine tune the setting for the largest packet size allowed. Open the SFS2X/config/core.xml file and edit the <maxIncomingRequestSize> parameter. By default it should be 2MB, which is pretty large. Typically game requests are much smaller than that, in the rage of a few KBytes. You can set a much lower value (e.g. 20-40KB) and restart your server instance. This will make sure that all suspiciously large requests will be dropped without wasting time on them.

Let us know if it helps.

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
arodrigo
Posts: 24
Joined: 22 Jul 2011, 10:03
Location: Spain
Contact:

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby arodrigo » 28 Oct 2013, 11:05

The first time we try to let one server alive and move the users to another to see how long can be the thread at 100% but at the 24h aprox we kill it. All the times we make a Thread dump.

What do you recommends us to check before the restart of the server the next time we have the error?

We are going to change the maxIncomingRequestSize of the half of our servers in order to compare the behavior.

Thanks!
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby Lapo » 28 Oct 2013, 11:35

arodrigo wrote:The first time we try to let one server alive and move the users to another to see how long can be the thread at 100% but at the 24h aprox we kill it. All the times we make a Thread dump.

This is interesting. So you are saying that you have tried moving the users away from the instance that had the thread at 100% and the problem didn't go away?
Did you move all users within 24 hours?
What was the lowest CCU count?

What do you recommends us to check before the restart of the server the next time we have the error?

It would be interesting to know what is the CCU count and the incoming traffic.
Another thing to check is the server logs, see if you can find any WARN level message that occurs frequently, especially in the period of time where the server is showing the problem.

We are going to change the maxIncomingRequestSize of the half of our servers in order to compare the behavior.

Good, let me know what happens.
Lapo

--

gotoAndPlay()

...addicted to flash games
ismaello
Posts: 5
Joined: 03 Jun 2013, 14:33

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby ismaello » 21 Feb 2014, 14:06

Hi Lapo,

The thread issue is still happening. At the moment when it happend, CCU: 750.

If you run netstat -anp, you can check all S.O.'s socket state to smartfox (port 443) are: CLOSE_WAIT.

You can do "telnet server_smartfox 443" and the connections work right, it's accepted, but the server doesn't work and the connection from my IP in the server is CLOSE_WAIT

So, all users are using Bluebox.

The value of maxIncomingRequestSize was 500000 in the last issue. We have changed it to 100000.

Regards.
grhwood
Posts: 116
Joined: 15 Mar 2011, 04:43

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby grhwood » 21 Feb 2014, 14:42

I have the CLOSE_WAIT issue today too
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Random infinite loop in java.util.zip.Inflater.inflateBy

Postby Lapo » 21 Feb 2014, 15:00

100K is still a pretty large value. Keep in mind that 100K is the max amount of data that the client can send and it is compressed. Compression can add a factor of 3x to 10x.

Do you really need very large requests to be sent by the clients? Typically clients will send very small packets in the range of 50-250 bytes with occasional spikes in the few KBytes. I would probably reduce the size to 10K and see if it reduces the incidence of the problem.

In case the issue appears again try taking multiple thread dumps every 10-15 minutes (at least 4-5 dumps) before restarting the server, then zip the files and send them over to our support@... email box. Please also add a reference to this discussion.

If you run netstat -anp, you can check all S.O.'s socket state to smartfox (port 443) are: CLOSE_WAIT.

CLOSE_WAIT simply tells us that clients have closed the connection but the Server hasn't replied yet with the corresponding FIN reply. Technically the server could still send data to the client although the latter wouldn't be able to receive it.

Why would all clients have closed the connection? Sounds strange. Smells like network problems.
Is there any firewall in front of the server that might be causing the drop?
Lapo

--

gotoAndPlay()

...addicted to flash games
NingLi
Posts: 1
Joined: 01 Nov 2016, 03:46

Re: Random infinite loop in java.util.zip.Inflater.inflateBytes

Postby NingLi » 01 Nov 2016, 04:21

hello,My English isbroken

I have a same question while receiving net packages .if you have fix the question .

Do you have any idea while deal with the problem.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Random infinite loop in java.util.zip.Inflater.inflateBytes

Postby Lapo » 01 Nov 2016, 11:51

NingLi wrote:hello,My English isbroken

I have a same question while receiving net packages .if you have fix the question .

Do you have any idea while deal with the problem.

HI,
it sounds like you are using a very old version of the server. I would highly recommend to at least update to version 2.9:
http://smartfoxserver.com/download/sfs2x#p=installer

Once you have done that you can also apply the latest patch 2.11.1
http://smartfoxserver.com/download/sfs2x#p=updates

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: Google [Bot] and 62 guests