Websocket Exception

[snehil89]

Hi there,

We have been getting a lot of
"19 Apr 2019 | 19:13:04,268 | WARN | SFSWorker:Ext:3 | sfs2x.websocket.SFS2XWSService | | I/O Exception writing byte[] to websocket: org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED], { Id: 949119, Type: WEBSOCKET, Logged: Yes, IP: 110.224.56.28:3388 }
"
upgraded to latest 2.13.5. using websocket binary protocol. memory usage (including native) is ever increasing ,finally have to restart server after 10-12 days.

In the heap dump can see a lot of websocketsessions open. What could be the possible cause ?

[Lapo]

Hi,
what is your average CCU on the server, and how many of these errors are you getting per hour or day?

Usually these errors are likely due to client side connection issues. What client type does your game target? Browser? Mobile?
I would also make sure that there are no bandwidth or network issues on your hosting side.

As regards the ever increasing memory that might be a different problem. More likely related with custom Extension code.

Let us know.

[snehil89]

Hi Lapo,

Normally 600 CCU, peak time 1200-1500 CCU. On an average 10 exceptions seen per hour. peak hours can go upto 70-80 as well.

Game targets javascript client over browser and mobile browsers (web-app) as well as we have native mobile apps for android and ios (C#) as well.

The reason I am relating exception to memory leakage is because as seen from dump, there are many jetty.websocket.* instances persistent in memory including websocketsession (attached is screenshot).

The things we changed while migration(2.9.2 -> 2.13.4) is sending json format data to javascript clients with wsbinary protocol on. Other wise no change on custom extension code.

[Lapo]

Hi,
the problem is that we can't say if those objects are leaking or not, because of the non-deterministic nature of the GC.
To get more evidence of it, one should be monitoring over the course of many days to track those objects down and see what happens.

One of your colleagues told us via email that you need to restart the server approximately every two weeks. Can you describe what happens after that time? Is the server crashing? If so do you have more details as to what is the cause?
Did you check the logs?

Are you monitoring the state of the heap during the two weeks? Do you see a an ever growing trend?
Or is it a problem with non-heap memory?

Thanks

[snehil89]

Hi Lapo,

The concern is that the memory used by the jvm process reaches above 15 GB (seen from top utility) in 12 days, however heap and meta space defined for java process is 4GB each and used memory as seen from gc logs and jstat utillity is 3.6GB and 1GB respectively.

So the question is - where is this native memory build up coming from ? In earlier version of smartfox (<=2.9.2) we never saw such increment in memory used by jvm process .

This increment in memory can certainly lead to OOM some day, if we don't restart.

And of course we have been tracking heap dump, I can see that websocketsession instance created on very first day still persistent in memory. Is it some how linked to jetty-9.3.11 websocket native memory leakage ? Do you have any idea about that ?

[Lapo]

It looks like it's a Jetty issue. Very likely it is this one:
https://github.com/eclipse/jetty.project/issues/293

We'll have to investigate further and see what possible solution we can implement.
I'll post an update when we have a clearer picture.

Stay tuned.

[Lapo]

We have addressed the issue using one of the solutions provided by J.Erdfelt in the same article (he’s one of the Jetty lead developers)
Specifically we have unregistered the “permessage-deflate” option in the WebsocketFactory which seems the cause of the native memory leak.

I've sent an email to your colleague with an attached .war file to replace in your current installation.

Let us know if it helps. If it does we’ll publish an official update.
Cheers