WebSocket binary message max size

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

moris
Posts: 52
Joined: 17 Aug 2012, 11:11

WebSocket binary message max size

Postby moris » 18 Oct 2018, 19:20

Hi

Recently I came across a problem with sending the big data to the server via WebSocket.

Code: Select all

org.eclipse.jetty.websocket.api.MessageTooLargeException: Binary message size [115773] exceeds maximum size [65536]
   at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidBinaryMessageSize(WebSocketPolicy.java:128)
   at org.eclipse.jetty.websocket.common.Parser.assertSanePayloadLength(Parser.java:130)
   at org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:480)
   at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:252)
   at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:679)
   at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
   at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   at java.lang.Thread.run(Thread.java:745)


Core.xml configuration

Code: Select all

<coreSettings>
   <systemControllerClass>com.smartfoxserver.v2.controllers.v290.SystemReqController</systemControllerClass>
   <extensionControllerClass>com.smartfoxserver.v2.controllers.v290.ExtensionReqController</extensionControllerClass>
   <ioHandlerClass>com.smartfoxserver.v2.protocol.DefaultIoHandler</ioHandlerClass>
   <sessionManagerClass>custom.class</sessionManagerClass>
   <packetQueuePolicyClass>custom.class</packetQueuePolicyClass>
   <readBufferType>Heap</readBufferType>
   <writeBufferType>Heap</writeBufferType>
   
   <!-- Editable settings -->
   
   <!-- Max size of incoming packet, in bytes -->
   <maxIncomingRequestSize>5000000</maxIncomingRequestSize>
   
   <!-- Max Size for read buffer -->
   <maxReadBufferSize>1024</maxReadBufferSize>
   
   <!-- Max Size for write buffer -->
   <maxWriteBufferSize>32768</maxWriteBufferSize>
   
   <!-- Session Queue Size -->
   <sessionPacketQueueSize>150</sessionPacketQueueSize>
   
   <!-- Thread Pool Size of the 3 main stages of the Bitswarm Engine -->
   <socketAcceptorThreadPoolSize>1</socketAcceptorThreadPoolSize>
   <socketReaderThreadPoolSize>1</socketReaderThreadPoolSize>
   <socketWriterThreadPoolSize>3</socketWriterThreadPoolSize>   
   
   <!-- Enable disable Nagle algorithm on sockets, true == disable -->
   <tcpNoDelay>true</tcpNoDelay>

   <!-- BlueBox Settings -->
   <bbMaxLogFiles>10</bbMaxLogFiles>
   <bbMaxLogFileSize>5000000</bbMaxLogFileSize>
   <bbDebugMode>false</bbDebugMode>
</coreSettings>
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: WebSocket binary message max size

Postby Lapo » 19 Oct 2018, 08:02

Hi,
interesting, looks like there's a limit within Jetty that we didn't know about.
We'll try to reproduce it locally and see how it can be increased. A quick search in Jetty's doc gave us a couple of hints on how to do it.

Will post an update soon.
Lapo
--
gotoAndPlay()
...addicted to flash games
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: WebSocket binary message max size

Postby Lapo » 19 Oct 2018, 16:27

UPDATE:
we were able to reproduce the issue and have fixed the problem by making sure that the "maxMessageSize" and "maxMessageBufferSize" are consistent with the maxIncomingRequestSize parameter under config/core.xml

We'll release a patch the next week with the update. If it's urgent you can send us an email to support@... and we'll send the files right away.

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
moris
Posts: 52
Joined: 17 Aug 2012, 11:11

Re: WebSocket binary message max size

Postby moris » 19 Oct 2018, 18:32

thanks for such a fast reply. It isn't urgent, I'll wait for the patch
User avatar
Rob
Posts: 53
Joined: 01 Jul 2017, 07:33

Re: WebSocket binary message max size

Postby Rob » 24 Oct 2018, 13:26

Not sure if this is related but I'm also running into message size problems when sending larger messages.

Getting a BufferOverflowException when sending a SFSObject with 350 keys. Sending a message with half of those keys works.

(I tried changing the maxWriteBufferSize but that didn't help)

Code: Select all

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Exception: java.nio.BufferOverflowException
Message: *** Null ***
Description: Error during websocket packet write
+--- --- ---+
Stack Trace:
+--- --- ---+
java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
java.nio.ByteBuffer.put(ByteBuffer.java:859)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.encodeSFSObjectKey(DefaultSFSDataSerializer.java:1382)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.obj2bin(DefaultSFSDataSerializer.java:516)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.object2binary(DefaultSFSDataSerializer.java:502)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.encodeObject(DefaultSFSDataSerializer.java:848)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.obj2bin(DefaultSFSDataSerializer.java:519)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.object2binary(DefaultSFSDataSerializer.java:502)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.encodeObject(DefaultSFSDataSerializer.java:848)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.obj2bin(DefaultSFSDataSerializer.java:519)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.object2binary(DefaultSFSDataSerializer.java:502)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.encodeObject(DefaultSFSDataSerializer.java:848)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.obj2bin(DefaultSFSDataSerializer.java:519)
com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.object2binary(DefaultSFSDataSerializer.java:502)
com.smartfoxserver.v2.entities.data.SFSObject.toBinary(SFSObject.java:234)
com.smartfoxserver.bitswarm.websocket.jetty.WebSocketBinaryProtocolCodec.onPacketWrite(WebSocketBinaryProtocolCodec.java:99)
com.smartfoxserver.bitswarm.core.BitSwarmEngine.writeToWebSocket(BitSwarmEngine.java:417)
com.smartfoxserver.bitswarm.core.BitSwarmEngine.write(BitSwarmEngine.java:398)
com.smartfoxserver.bitswarm.io.Response.write(Response.java:71)
com.smartfoxserver.v2.api.SFSApi.login(SFSApi.java:547)
com.smartfoxserver.v2.controllers.system.Login.execute(Login.java:244)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.executeEventCommand(SFSExtensionManager.java:1036)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.dispatchEvent(SFSExtensionManager.java:776)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.dispatchZoneLevelEvent(SFSExtensionManager.java:689)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.handleServerEvent(SFSExtensionManager.java:887)
com.smartfoxserver.v2.core.SFSEventManager$SFSEventRunner.run(SFSEventManager.java:66)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: WebSocket binary message max size

Postby Lapo » 24 Oct 2018, 14:10

Hi,
this seems a different beast. Can you show me exactly how this problem can be reproduced?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Rob
Posts: 53
Joined: 01 Jul 2017, 07:33

Re: WebSocket binary message max size

Postby Rob » 24 Oct 2018, 15:24

Lapo wrote:Can you show me exactly how this problem can be reproduced?

I assume it's just the size of the message since it works up to a certain number of keys in the SFSObject. But in my particular case I'm looping over a database result with 350 records and adding SFSArrays to an SFSObject like:

Code: Select all

SFSObject lpl = new SFSObject();
while (rset.next()) {
   SFSArray a = new SFSArray();
   a.addByte((byte) area);
   a.addByte((byte) type);
   a.addShort((short) id);
   a.addUtfString(desc);
   lpl.putSFSArray(name, a);
}

name is a string ~20 characters and desc is a string up to ~100 characters.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: WebSocket binary message max size

Postby Lapo » 25 Oct 2018, 10:35

Hi,
I can't reproduce it. I tried replicating the same data structure you're using and I can send thousands of objects without problem.

I suspect the cause of the problem is that a null value is added to the SFSArray while cycling through the database result. I would suggest to add null checks to avoid potential runtime serialization issues.

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Rob
Posts: 53
Joined: 01 Jul 2017, 07:33

Re: WebSocket binary message max size

Postby Rob » 25 Oct 2018, 12:41

Lapo wrote:I suspect the cause of the problem is that a null value is added to the SFSArray while cycling through the database result. I would suggest to add null checks to avoid potential runtime serialization issues.

I added checks for everything added to the SFSObject and it turns out the cause of the problem is that some names are Turkish and SFS doesn't like object keys with foreign characters. So doing something simple as this will cause the BufferOverflowException:

Code: Select all

SFSObject o = new SFSObject();
o.putInt("Türkçe", 0);
send("TEST", o, user);
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: WebSocket binary message max size

Postby Lapo » 25 Oct 2018, 14:36

Ah yes.
Keys in an SFSObject are not multi-byte. In other words UTF is not supported for SFSObject keys.
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Rob
Posts: 53
Joined: 01 Jul 2017, 07:33

Re: WebSocket binary message max size

Postby Rob » 25 Oct 2018, 18:42

Lapo wrote:UTF is not supported for SFSObject keys.

Aha, maybe this limitation should be documented. I haven't seen any mention of this before.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: WebSocket binary message max size

Postby Lapo » 25 Oct 2018, 19:07

You're right. We'll update the docs.
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 35 guests