Our production server keeps crashing every 7~10 days due to OutOfMemoryError. I quite believe there's a memory leak whenever we receive following exception
04:12:42,302 WARN [pool-1-thread-3] websocket.SFS2XWSService - I/O Exception writing text to websocket: org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED], { Id: 12, Type: WEBSOCKET, Logged: Yes, IP: 14.169.41.202:58427 }
04:13:14,901 WARN [pool-1-thread-2] compress.CompressExtension -
java.io.EOFException: Connection has been closed locally
at org.eclipse.jetty.websocket.common.io.FrameFlusher.enqueue(FrameFlusher.java:339)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:614)
at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:182)
at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextOutgoingFrame(PerMessageDeflateExtension.java:116)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.access$1000(CompressExtension.java:42)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.compress(CompressExtension.java:534)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.deflate(CompressExtension.java:435)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.process(CompressExtension.java:415)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.outgoingFrame(CompressExtension.java:218)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack$Flusher.process(ExtensionStack.java:395)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:296)
at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.uncheckedSendFrame(WebSocketRemoteEndpoint.java:310)
at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:240)
at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:412)
at sfs2x.websocket.SFS2XWSService$WSChannelListener.write(SFS2XWSService.java:369)
at com.smartfoxserver.bitswarm.websocket.jetty.WebSocketTextProtocolCodec.onPacketWrite(WebSocketTextProtocolCodec.java:106)
at com.smartfoxserver.bitswarm.core.BitSwarmEngine.writeToWebSocket(BitSwarmEngine.java:417)
at com.smartfoxserver.bitswarm.core.BitSwarmEngine.write(BitSwarmEngine.java:398)
at com.smartfoxserver.bitswarm.io.Response.write(Response.java:71)
at com.smartfoxserver.v2.api.response.SFSResponseApi.sendExtResponse(SFSResponseApi.java:91)
at com.smartfoxserver.v2.api.SFSApi.sendExtensionResponse(SFSApi.java:1515)
at com.smartfoxserver.v2.extensions.BaseSFSExtension.send(BaseSFSExtension.java:487)
at com.smartfoxserver.v2.extensions.BaseSFSExtension.send(BaseSFSExtension.java:462)
at com.exoty.event.EventHandler.test(EventHandler.java:416)
at com.exoty.event.EventHandler.lambda$test$1(EventHandler.java:420)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
04:13:52,447 WARN [qtp1638472859-62] compress.CompressExtension -
java.nio.channels.ClosedChannelException
at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:498)
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:113)
at org.eclipse.jetty.io.ChannelEndPoint.close(ChannelEndPoint.java:118)
at org.eclipse.jetty.io.SelectChannelEndPoint.close(SelectChannelEndPoint.java:276)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:935)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:311)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.access$300(AbstractWebSocketConnection.java:61)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$OnDisconnectCallback.writeFailed(AbstractWebSocketConnection.java:101)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.notifyCallbackFailure(FrameFlusher.java:391)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.enqueue(FrameFlusher.java:339)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:614)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onConnectionStateChange(AbstractWebSocketConnection.java:473)
at org.eclipse.jetty.websocket.common.io.IOState.notifyStateListeners(IOState.java:184)
at org.eclipse.jetty.websocket.common.io.IOState.onReadFailure(IOState.java:498)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:666)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.base/java.lang.Thread.run(Thread.java:844)
04:15:42,300 WARN [qtp1638472859-45] compress.CompressExtension -
java.io.EOFException: Connection has been closed locally
at org.eclipse.jetty.websocket.common.io.FrameFlusher.enqueue(FrameFlusher.java:339)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:614)
at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:182)
at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextOutgoingFrame(PerMessageDeflateExtension.java:116)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.access$1000(CompressExtension.java:42)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.compress(CompressExtension.java:534)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.deflate(CompressExtension.java:435)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.process(CompressExtension.java:415)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.succeeded(IteratingCallback.java:365)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.writeSuccess(CompressExtension.java:557)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.notifyCallbackSuccess(FrameFlusher.java:407)
at org.eclipse.jetty.websocket.common.io.FrameFlusher$Flusher.succeedEntries(FrameFlusher.java:242)
at org.eclipse.jetty.websocket.common.io.FrameFlusher$Flusher.succeeded(FrameFlusher.java:232)
at org.eclipse.jetty.io.WriteFlusher$PendingState.complete(WriteFlusher.java:269)
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:394)
at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:101)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint$1.succeeded(SslConnection.java:292)
at org.eclipse.jetty.io.WriteFlusher$PendingState.complete(WriteFlusher.java:269)
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:394)
at org.eclipse.jetty.io.SelectChannelEndPoint$3.run(SelectChannelEndPoint.java:107)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.base/java.lang.Thread.run(Thread.java:844)
I've tried to reproduce this issue on our staging environment with following code
Code: Select all
private void handleUserJoinZone(ISFSEvent evt) {
....
if (user.getSession().getType() == SessionType.WEBSOCKET) {
SmartFoxServer.getInstance().getTaskScheduler().schedule(() -> this.test(user), 1, TimeUnit.SECONDS);
}
}
private void test(User user) {
List<Integer> list = IntStream.range(0, 1000).boxed().collect(Collectors.toList());
ISFSObject data = new SFSObject();
data.putIntArray("t", list);
for (int i = 0; i < 1000; i++) {
extension.send("hoang", data, user);
}
if (user.isConnected()) {
SmartFoxServer.getInstance().getTaskScheduler().schedule(() -> this.test(user), 1, TimeUnit.SECONDS);
}
}
I close browser tab to generate the exception. Please take a look at attachment for the heap dump.
SmartfoxServer 2X v2.13
CentOS 7
JDK 9.0.1
I'm not sure if this issue is same as https://stackoverflow.com/questions/34578235/jetty-possible-memory-leak-when-using-websockets-and-bytebuffer
I've tried to upgrade jetty to Release 9.4.8.v20171121 but failed (cannot make webservice works).
So, what do you think about this issue? And how can I upgrade jetty to its latest version?
Thanks,
Hoang