Internal Error

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Internal Error

Postby Mrm83 » 14 May 2019, 02:44

Just saw this in the logs..

13 May 2019 | 12:49:50,951 | WARN | qtp1814908102-305 | util.thread.QueuedThreadPool | |
java.lang.IllegalStateException: Internal error
at sun.security.ssl.SSLEngineImpl.initHandshaker(SSLEngineImpl.java:470)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1007)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:525)
at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:325)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:233)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
13 May 2019 | 12:49:50,952 | WARN | qtp1814908102-305 | util.thread.QueuedThreadPool | | Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@583e1fb3 in qtp1814908102{STARTED,10<=10<=200,i=5,q=0}
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 14 May 2019, 08:20

Hi,
never seen this before. Do you have many of these errors in the logs, or just one instance?
It looks like Jetty had a problem with an SSL transaction, could be a bad request coming from a client trying to send malformed data.

What version of SFS2X is this?
Let us know
Lapo
--
gotoAndPlay()
...addicted to flash games
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 14 May 2019, 12:34

Just noticed that 1 in the daily log file, I havn't checked previous days.

This is from SFS 2.13.0

The error doesn't say where in my classes it happened..
Is there a way to catch that error when it occurs so I can log additional info such as username?
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 14 May 2019, 13:04

The server also crashed yesterday, I dont see any errors in the log.

What could cause the server to crash?? This is the first time it happened in half a year.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 14 May 2019, 13:50

Mrm83 wrote:Just noticed that 1 in the daily log file, I havn't checked previous days.

This is from SFS 2.13.0

The error doesn't say where in my classes it happened..
Is there a way to catch that error when it occurs so I can log additional info such as username?

That's not happening in your classes, it's happening in Jetty (the web server) when someone is starting an SSL transaction. It's not possible to know what user caused it because it's too early in the process.

What could cause the server to crash?? This is the first time it happened in half a year.

Did you check the logs at the time of the crash?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 14 May 2019, 14:47

when you say "someone is starting an SSL transaction", is that at connection to the server?

Yes, I've checked smartfox.log as well as my other log files, no exception, no fatal/errors.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 15 May 2019, 07:47

Mrm83 wrote:when you say "someone is starting an SSL transaction", is that at connection to the server?

Yes, to the web server using HTTPS.
The exception seems to indicate an error in the initial SSL handshake.

Yes, I've checked smartfox.log as well as my other log files, no exception, no fatal/errors.

So what do you mean by crash, exactly?
Did the SFS2X process die? Or was it still running but not responding? Something else?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 15 May 2019, 14:22

The sfs instance restarted by itself.

This could only happen due to crash right? The actual server did not reboot.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 15 May 2019, 14:57

If the process was restarted it could be a crash but also it could be the OOM-Killer, at least if your OS is Linux.
If you're running under Linux the kernel may decide to kill a specific process if necessary, when the system is running low on memory.

You can check your server logs, typically under /var/log/syslog or /var/log/messages you can find a message along these lines:

Code: Select all

... kernel: Out of Memory: Killed process XYZ (java)


Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 15 May 2019, 19:33

The server is running un Ubuntu.

Youre right, it seems it did run out of memory from the kernal log. But it killed mysqld and not sfs. Could killing mysqld cause SFS to reboot?

This is very strange because it should not have ran out of memory.
Is there a way to debug this?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 16 May 2019, 07:11

Mrm83 wrote:The server is running un Ubuntu.

Youre right, it seems it did run out of memory from the kernal log. But it killed mysqld and not sfs. Could killing mysqld cause SFS to reboot?

No, but if your system was running low on resources SFS2X might have died due to a lack of memory.
Typically you should find traces of an out-of-memory crash in the logs themselves, but if the cause of the crash was different the logs won't help.
Also the JVM typically creates a hs_err_pid file in the main application folder (SFS2X/ in this case). Did find any?

This is very strange because it should not have ran out of memory.
Is there a way to debug this?

Is the system running in the cloud? If so how much RAM does the system have?
Is your OS configured to use a swap file or swap partitions?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 16 May 2019, 14:01

There is no hs_err_pid file

Yes, it is in the cloud. It has 1G ram, max CCU is under 20. Memory allocated is consistently at 160mb.
It is configured to use swap partition, size is currently at 256MB.

Sys stats:
System load: 0.11
Memory usage: 65%
Swap usage: 57%
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 16 May 2019, 14:48

Thanks,
with a CCU < 20 you should never see any issues or crashes. Unless there's some issue with your server side code.

The swap file size doesn't sound too good as typically you would want 1-2GB of swap for a 1GB RAM system. Maybe fixing this will help stabilizing memory usage.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
Mrm83
Posts: 155
Joined: 17 Dec 2017, 04:02

Re: Internal Error

Postby Mrm83 » 16 May 2019, 15:13

Yes thats what i thought. With 20 CCU it shouldnt crash but it did run out of memory.

This is the first time it happend in 6 months.

Right, the current server spec is in synced with our internal testing limits. This helps us see potential issues at its current scaling.
So changing it 1+ gb will definitely resolve the issue because it is way above the limit, but it doesn't really solve the inherit problem as it currently crashed below limit.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Internal Error

Postby Lapo » 16 May 2019, 16:05

Mrm83 wrote:Yes thats what i thought. With 20 CCU it shouldnt crash but it did run out of memory.

I wouldn't be so sure about that. I don't think you have found evidence for a memory problem in SFS2X.
The MySQL restart may or may not be related but it doesn't tell us what happened to SFS2X. Also OutOfMemory errors are expected in the logs, if there's not a trace of them it's hard to say what really happened.

Since there are no crash dump files either, you may want to add an extra JVM config parameter to specify where said dump file should be written to: https://docs.oracle.com/javase/8/docs/t ... og001.html

Goes without saying: make sure the location has write permissions enabled :)

Hope it helsp
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: Thomasea and 50 guests