Just saw this in the logs..
13 May 2019 | 12:49:50,951 | WARN | qtp1814908102-305 | util.thread.QueuedThreadPool | |
java.lang.IllegalStateException: Internal error
at sun.security.ssl.SSLEngineImpl.initHandshaker(SSLEngineImpl.java:470)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1007)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:525)
at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:325)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:233)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
13 May 2019 | 12:49:50,952 | WARN | qtp1814908102-305 | util.thread.QueuedThreadPool | | Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@583e1fb3 in qtp1814908102{STARTED,10<=10<=200,i=5,q=0}
Internal Error
Re: Internal Error
Hi,
never seen this before. Do you have many of these errors in the logs, or just one instance?
It looks like Jetty had a problem with an SSL transaction, could be a bad request coming from a client trying to send malformed data.
What version of SFS2X is this?
Let us know
never seen this before. Do you have many of these errors in the logs, or just one instance?
It looks like Jetty had a problem with an SSL transaction, could be a bad request coming from a client trying to send malformed data.
What version of SFS2X is this?
Let us know
Re: Internal Error
Just noticed that 1 in the daily log file, I havn't checked previous days.
This is from SFS 2.13.0
The error doesn't say where in my classes it happened..
Is there a way to catch that error when it occurs so I can log additional info such as username?
This is from SFS 2.13.0
The error doesn't say where in my classes it happened..
Is there a way to catch that error when it occurs so I can log additional info such as username?
Re: Internal Error
The server also crashed yesterday, I dont see any errors in the log.
What could cause the server to crash?? This is the first time it happened in half a year.
What could cause the server to crash?? This is the first time it happened in half a year.
Re: Internal Error
Mrm83 wrote:Just noticed that 1 in the daily log file, I havn't checked previous days.
This is from SFS 2.13.0
The error doesn't say where in my classes it happened..
Is there a way to catch that error when it occurs so I can log additional info such as username?
That's not happening in your classes, it's happening in Jetty (the web server) when someone is starting an SSL transaction. It's not possible to know what user caused it because it's too early in the process.
What could cause the server to crash?? This is the first time it happened in half a year.
Did you check the logs at the time of the crash?
Thanks
Re: Internal Error
when you say "someone is starting an SSL transaction", is that at connection to the server?
Yes, I've checked smartfox.log as well as my other log files, no exception, no fatal/errors.
Yes, I've checked smartfox.log as well as my other log files, no exception, no fatal/errors.
Re: Internal Error
Mrm83 wrote:when you say "someone is starting an SSL transaction", is that at connection to the server?
Yes, to the web server using HTTPS.
The exception seems to indicate an error in the initial SSL handshake.
Yes, I've checked smartfox.log as well as my other log files, no exception, no fatal/errors.
So what do you mean by crash, exactly?
Did the SFS2X process die? Or was it still running but not responding? Something else?
Thanks
Re: Internal Error
The sfs instance restarted by itself.
This could only happen due to crash right? The actual server did not reboot.
This could only happen due to crash right? The actual server did not reboot.
Re: Internal Error
If the process was restarted it could be a crash but also it could be the OOM-Killer, at least if your OS is Linux.
If you're running under Linux the kernel may decide to kill a specific process if necessary, when the system is running low on memory.
You can check your server logs, typically under /var/log/syslog or /var/log/messages you can find a message along these lines:
Hope it helps
If you're running under Linux the kernel may decide to kill a specific process if necessary, when the system is running low on memory.
You can check your server logs, typically under /var/log/syslog or /var/log/messages you can find a message along these lines:
Code: Select all
... kernel: Out of Memory: Killed process XYZ (java)
Hope it helps
Re: Internal Error
The server is running un Ubuntu.
Youre right, it seems it did run out of memory from the kernal log. But it killed mysqld and not sfs. Could killing mysqld cause SFS to reboot?
This is very strange because it should not have ran out of memory.
Is there a way to debug this?
Youre right, it seems it did run out of memory from the kernal log. But it killed mysqld and not sfs. Could killing mysqld cause SFS to reboot?
This is very strange because it should not have ran out of memory.
Is there a way to debug this?
Re: Internal Error
Mrm83 wrote:The server is running un Ubuntu.
Youre right, it seems it did run out of memory from the kernal log. But it killed mysqld and not sfs. Could killing mysqld cause SFS to reboot?
No, but if your system was running low on resources SFS2X might have died due to a lack of memory.
Typically you should find traces of an out-of-memory crash in the logs themselves, but if the cause of the crash was different the logs won't help.
Also the JVM typically creates a hs_err_pid file in the main application folder (SFS2X/ in this case). Did find any?
This is very strange because it should not have ran out of memory.
Is there a way to debug this?
Is the system running in the cloud? If so how much RAM does the system have?
Is your OS configured to use a swap file or swap partitions?
Thanks
Re: Internal Error
There is no hs_err_pid file
Yes, it is in the cloud. It has 1G ram, max CCU is under 20. Memory allocated is consistently at 160mb.
It is configured to use swap partition, size is currently at 256MB.
Sys stats:
System load: 0.11
Memory usage: 65%
Swap usage: 57%
Yes, it is in the cloud. It has 1G ram, max CCU is under 20. Memory allocated is consistently at 160mb.
It is configured to use swap partition, size is currently at 256MB.
Sys stats:
System load: 0.11
Memory usage: 65%
Swap usage: 57%
Re: Internal Error
Thanks,
with a CCU < 20 you should never see any issues or crashes. Unless there's some issue with your server side code.
The swap file size doesn't sound too good as typically you would want 1-2GB of swap for a 1GB RAM system. Maybe fixing this will help stabilizing memory usage.
Cheers
with a CCU < 20 you should never see any issues or crashes. Unless there's some issue with your server side code.
The swap file size doesn't sound too good as typically you would want 1-2GB of swap for a 1GB RAM system. Maybe fixing this will help stabilizing memory usage.
Cheers
Re: Internal Error
Yes thats what i thought. With 20 CCU it shouldnt crash but it did run out of memory.
This is the first time it happend in 6 months.
Right, the current server spec is in synced with our internal testing limits. This helps us see potential issues at its current scaling.
So changing it 1+ gb will definitely resolve the issue because it is way above the limit, but it doesn't really solve the inherit problem as it currently crashed below limit.
This is the first time it happend in 6 months.
Right, the current server spec is in synced with our internal testing limits. This helps us see potential issues at its current scaling.
So changing it 1+ gb will definitely resolve the issue because it is way above the limit, but it doesn't really solve the inherit problem as it currently crashed below limit.
Re: Internal Error
Mrm83 wrote:Yes thats what i thought. With 20 CCU it shouldnt crash but it did run out of memory.
I wouldn't be so sure about that. I don't think you have found evidence for a memory problem in SFS2X.
The MySQL restart may or may not be related but it doesn't tell us what happened to SFS2X. Also OutOfMemory errors are expected in the logs, if there's not a trace of them it's hard to say what really happened.
Since there are no crash dump files either, you may want to add an extra JVM config parameter to specify where said dump file should be written to: https://docs.oracle.com/javase/8/docs/t ... og001.html
Goes without saying: make sure the location has write permissions enabled
Hope it helsp
Who is online
Users browsing this forum: Thomasea and 50 guests