I recently encountered the below mentioned exception while running JAVA client api to test load on smartfox server:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.<init>(SocketSendBufferPool.java:159)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool.<init>(SocketSendBufferPool.java:46)
at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:84)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.<init>(NioClientSocketPipelineSink.java:74)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:135)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:105)
at sfs2x.client.core.sockets.TCPSocketLayer.initNetty(TCPSocketLayer.java:84)
at sfs2x.client.core.sockets.TCPSocketLayer.connect(TCPSocketLayer.java:195)
at sfs2x.client.bitswarm.BitSwarmClient.connect(BitSwarmClient.java:222)
at sfs2x.client.bitswarm.BitSwarmClient.reconnect(BitSwarmClient.java:382)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketError(BitSwarmClient.java:445)
at sfs2x.client.bitswarm.BitSwarmClient.access$3(BitSwarmClient.java:441)
at sfs2x.client.bitswarm.BitSwarmClient$4.dispatch(BitSwarmClient.java:116)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnError(TCPSocketLayer.java:168)
at sfs2x.client.core.sockets.TCPSocketLayer.handleErrorThread(TCPSocketLayer.java:129)
at sfs2x.client.core.sockets.TCPSocketLayer.access$8(TCPSocketLayer.java:121)
at sfs2x.client.core.sockets.TCPSocketLayer$1.run(TCPSocketLayer.java:113)
at java.lang.Thread.run(Thread.java:745)
Upon checking the mentioned exception, I found there is a bug in netty related to memory leak [issue][/https://issues.jboss.org/browse/NETTY-424?_sscc=t].
There is also a github issue posted for the same [github-issue][/https://github.com/netty/netty/issues/1574].
Can you just check if the above issue is present in the smartfox server also because some users experienced the same issue after netty 4.0. and smartfox 2.9.2 and 2.12.1 use netty-3.9.0.Final?
JAVA client api version : 1.1.0
Smartfox server version : 2.9.2
OS : Centos 6.5
java.lang.OutOfMemoryError: Direct buffer memory
Re: java.lang.OutOfMemoryError: Direct buffer memory
SmartFoxServer 2X runs it's own socket engine. We only use Netty for websockets and we have no reports or evidence of any issues on the server side.
The error you have shown is related to the client side API. Maybe you're pushing too many instances of the client in the same JVM?
Thanks
The error you have shown is related to the client side API. Maybe you're pushing too many instances of the client in the same JVM?
Thanks
Re: java.lang.OutOfMemoryError: Direct buffer memory
Thank you for responding.
I am running 1000 users in a single java process with the below mentioned parameters:
java -Xmx3072M -XX:MaxPermSize=128M -Xss256k -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:-HeapDumpOnOutOfMemoryError
I tried to run 3 such processes on a system and I am using 2 systems to create the load, but somehow all the users related to a process disconnect at the same time. I tried to take the heap and thread dump but the process is unresponsive.
I read the 100K concurrent users white paper and thought we can try the same.
I am also getting the below mentioned exceptions:
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 5138
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: Unknow SFSDataType ID: -128
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:225)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSDataType. Expected: 18, found: 0
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:182)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 4608
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 4608
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
After 3 Hours, 14 Minutes:
Number of clients connected:7918
CPU Usage: 6.13%
Memory < 1 GB
Number of rooms: 1231
Transferred data: 33.95 GB out | 674.39 Mb in
Current Data transfer rate: OUT: 1.97 MB/s | IN: 141 KB/s
Avarage Data transfer rate: OUT: 2.91 MB/s | IN: 57.72 KB/s
Dropped Packets: Out: 448867 (0%) | In: 0
Just after this, one client process stops responding and all 1000 users disconnect. It raises the CPU and Memory on Smartfox server.
We are not sure whether this is due to large amount of data or the problem with the client code.
Do you have any idea about this. Can you share any problems faced while working on 100K users ?
Configuration of system on which smartfox server is installed:
version: 2.9.2
OS : Centos 6.5
8 cores
15 GB RAM
-Xmx4096M
Configuration of system on which clients are:
OS: Centos 6.5
8 cores
61 GB RAM
version: 1.1.0
I am running 1000 users in a single java process with the below mentioned parameters:
java -Xmx3072M -XX:MaxPermSize=128M -Xss256k -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:-HeapDumpOnOutOfMemoryError
I tried to run 3 such processes on a system and I am using 2 systems to create the load, but somehow all the users related to a process disconnect at the same time. I tried to take the heap and thread dump but the process is unresponsive.
I read the 100K concurrent users white paper and thought we can try the same.
I am also getting the below mentioned exceptions:
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 5138
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: Unknow SFSDataType ID: -128
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:225)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSDataType. Expected: 18, found: 0
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:182)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 4608
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 4608
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
After 3 Hours, 14 Minutes:
Number of clients connected:7918
CPU Usage: 6.13%
Memory < 1 GB
Number of rooms: 1231
Transferred data: 33.95 GB out | 674.39 Mb in
Current Data transfer rate: OUT: 1.97 MB/s | IN: 141 KB/s
Avarage Data transfer rate: OUT: 2.91 MB/s | IN: 57.72 KB/s
Dropped Packets: Out: 448867 (0%) | In: 0
Just after this, one client process stops responding and all 1000 users disconnect. It raises the CPU and Memory on Smartfox server.
We are not sure whether this is due to large amount of data or the problem with the client code.
Do you have any idea about this. Can you share any problems faced while working on 100K users ?
Configuration of system on which smartfox server is installed:
version: 2.9.2
OS : Centos 6.5
8 cores
15 GB RAM
-Xmx4096M
Configuration of system on which clients are:
OS: Centos 6.5
8 cores
61 GB RAM
version: 1.1.0
Re: java.lang.OutOfMemoryError: Direct buffer memory
Hi,
you won't be able to reach 100K users with this approach. We used a more sophisticated tool that still uses the API but replaces the socket engine with a much more light-weight one that doesn't create thousands of Netty's Channel Factories, thread pools etc... This in turn allows to keep resource consumption to a minimum, running a single Channel Factory and a single thread pool.
If you're running on Linux keep also in mind that when you approach the 1000 connection mark you will need to fine tune the "ulimit" parameter to allow for more file descriptors, as this is also a source of issues.
Try to reduce the number of clients per JVM until it runs stable. Then run one single JVM for a while (say 1 hour) and make sure there are no problems. Then you can start more JVMs until your CPU / RAM resources reach 90%, but make sure you don't max out either of those (cpu/ram) otherwise the test clients will start lagging behind and misbehave.
Cheers
you won't be able to reach 100K users with this approach. We used a more sophisticated tool that still uses the API but replaces the socket engine with a much more light-weight one that doesn't create thousands of Netty's Channel Factories, thread pools etc... This in turn allows to keep resource consumption to a minimum, running a single Channel Factory and a single thread pool.
If you're running on Linux keep also in mind that when you approach the 1000 connection mark you will need to fine tune the "ulimit" parameter to allow for more file descriptors, as this is also a source of issues.
I tried to run 3 such processes on a system and I am using 2 systems to create the load, but somehow all the users related to a process disconnect at the same time. I tried to take the heap and thread dump but the process is unresponsive.
Try to reduce the number of clients per JVM until it runs stable. Then run one single JVM for a while (say 1 hour) and make sure there are no problems. Then you can start more JVMs until your CPU / RAM resources reach 90%, but make sure you don't max out either of those (cpu/ram) otherwise the test clients will start lagging behind and misbehave.
Cheers
Re: java.lang.OutOfMemoryError: Direct buffer memory
Thank you for the help.
I tried the method you mentioned and did not face continuous disconnection since yesterday.
I have one more question:
When the server user count crossed 8000 (approx.) users. Around 1000 users disconnected, and the CPU utilization on Smartfox panel crossed 60% (its per core I guess). The system has 8 cores, that means it is 8*60=480%.
Is this normal for smartfox ? When you tested for 100K users, did you see the same CPU spike when users got disconnected?
I tried the method you mentioned and did not face continuous disconnection since yesterday.
I have one more question:
When the server user count crossed 8000 (approx.) users. Around 1000 users disconnected, and the CPU utilization on Smartfox panel crossed 60% (its per core I guess). The system has 8 cores, that means it is 8*60=480%.
Is this normal for smartfox ? When you tested for 100K users, did you see the same CPU spike when users got disconnected?
Re: java.lang.OutOfMemoryError: Direct buffer memory
Yes, massive disconnections at once will necessarily create a spike because the server has lots of work to do, clearing many data structures, sending updates to all remaining clients, resetting room and user variables etc...
Typically however, it is just a spike, so it will last for a very short time.
Typically however, it is just a spike, so it will last for a very short time.
Re: java.lang.OutOfMemoryError: Direct buffer memory
when User count 21891 Users were connected, there were 54 threads in the admin panel and after sometimes when around 19118 were disconnected, thread count reached to 94 after 2-3 minutes.
Is it normal for the server ?
Is it normal for the server ?
Re: java.lang.OutOfMemoryError: Direct buffer memory
Yes, the number of threads oscillates based on the load.
Disconnecting large number of clients at once is a lot of work for the server, cleaning up lots of data and sending updates to everyone else.
cheers
Disconnecting large number of clients at once is a lot of work for the server, cleaning up lots of data and sending updates to everyone else.
cheers
Who is online
Users browsing this forum: Stevenor and 90 guests