Client problems while load testing

[ers]

Hi folks,

We're having some problems while load testing the Smartfox server (PRO 1.6.6) with the Java Client API.

We're generating load from two separate client machines to a single server. Each client tries to create 200 virtual users that perform the following actions:

* Connect
* Login
* Join room
* Walk 10 times (involves a sfs.setUserVariables() and a sfs.sendObject() )
* Logout
* Disconnect

We're having about 110 failures (out of 400) in the join action and the server seems to be fine with no dropped messages.

We're using JMeter with custom samplers to orchestrate the load generation and the client machines are quad cores with 4 GB of RAM (2 GB assigned to the JMeter heap).

We have found some of these in the log files from both client machines:

05:33:54.239 [AnonymousIoService-5] WARN i.g.u.n.x.XMLSocketClientHandler - [/XXX.XXX.XXX.XXX:9339] EXCEPTION, please implement it.gotoandplay.utils.net.xmlsocket.XMLSocketClientHandler.exceptionCaught() for proper handling:
org.apache.mina.filter.codec.ProtocolDecoderException: org.apache.mina.common.BufferDataException: Line is too long: 33564 (Hexdump: 76 61 72 73 3E 3C 2F 76 61 72 73 3E 3C 2F 75 3E 3C 75 20 69 3D 27 31 37 33 27 20 6D 3D 27 30 27 3E 3C 6E 3E 3C 21 5B 43 44 41 54 41 5B 32 37 32 5D 5D 3E 3C 2F 6E 3E 3C 76 61 72 73 3E 3C 2F 76 61 72 73 3E 3C 2F 75 3E 3C 75 20 69 3D 27 31 37 32 27 20 6D 3D 27 30 27 3E 3C 6E 3E 3C 21 5B 43 44 41 54 41 5B 35 32 32 5D 5D 3E 3C 2F 6E 3E 3C 76 61 72 73 3E 3C 2F 76 61 72 73 3E 3C 2F 75 3E 3C 75 20 69 3D 27 31 37 34 27 20 6D 3D 27 30 27 3E 3C 6E 3E 3C 21 5B 43 44 41 54 41 5B 35 32 33 5D 5D 3E 3C 2F 6E 3E 3C 76 61 72 73 3E 3C 2F 76 61 72 73 3E 3C 2F 75 3E 3C 75 20 69 3D 27 31 37 35 27 20 6D 3D 27 30 27 3E 3C 6E 3E 3C 21 5B 43 44 41 54 41 5B 32 37 34 5D 5D 3E 3C 2F 6E 3E 3C 76 61 72 73 3E 3C 2F 76 61 72 73 3E 3C 2F 75 3E 3C 2F 75 4C 73 3E 3C 2F 62 6F 64 79 3E 3C 2F 6D 73 67 3E 00)

and lots of null pointer exceptions:

05:36:09.553 [AnonymousIoService-10] WARN i.g.u.n.x.XMLSocketClientHandler - [/XXX.XXX.XXX.XXX:9339] EXCEPTION, please implement it.gotoandplay.utils.net.xmlsocket.XMLSocketClientHandler.exceptionCaught() for proper handling:
java.lang.NullPointerException
at it.gotoandplay.smartfoxclient.handlers.SysHandler.handleUserLeaveRoom(SysHandler.java:473)

It seems to be a problem with the Smartfox API not being able to process the low level communication. Could you please help us in debugging it?

Thanks a lot in advance,

Esteban

[Lapo]

Hi,
it looks like the network layer of the API (based on Apache Mina) is complaining for a packet of data that exceeds its buffer capabilities. It looks like the buffer is limited around 32Kb and that's probably the origin of the problem.

I am not sure if you can re-configure that via code in the API itself. I will give it a look and get back to you

[dromos]

Strangely enough, I am also in the process of writing a load testing client with the SFS Java API and I believe I just ran into the exact same issue. My client successfully sends and receives dozens of messages but for some reason, when a large message is sent from the server, my client code never receives a notification for it. This has stopped me dead in my tracks and I'm not sure how to proceed.

[EDIT] - I just noticed this message was originally posted over a month ago. Did you get back to him through some other means or...?

[dromos]

For what it's worth, I verified that I am getting the same exception. Here's the stack trace...

EXCEPTION, please implement it.gotoandplay.utils.net.xmlsocket.XMLSocketClientHandler.exceptionCaught() for proper handling:
[java] org.apache.mina.filter.codec.ProtocolDecoderException: org.apache.mina.common.BufferDataException: Line is too long: 49497 (Hexdump: 69 66 69 63 61 74 69 6F 6E 20 74 6F 20 68 69 67 68 20 7A 6F 6F 6D 20 73 6E 69 70 65 72 20 6D 6F 64 65 6C 73 2E 22 2C 22 6E 61 6D 65 22 3A 22 5A 6F 72 69 6E 20 4F 70 74 69 63 73 20 53 63 6F 70 65 22 2C 22 72 61 72 69 74 79 22 3A 31 2C 22 74 68 75 6D 62 55 72 6C 22 3A 22 61 73 73 65 74 73 2F 69 63 6F 6E 73 2F 6C 61 62 2F 77 65 61 70 6F 6E 5F 63 6F 6D 70 6F 6E 65 6E 74 73 2F 6F 70 74 69 63 61 6C 5F 73 63 6F 70 65 2E 70 6E 67 22 2C 22 73 6B 75 22 3A 7B 22 77 65 61 6C 74 68 43 6F 73 74 22 3A 30 2C 22 73 74 61 74 69 6F 6E 43 61 73 68 43 6F 73 74 22 3A 30 2C 22 73 6B 75 49 64 22 3A 30 2C 22 74 79 70 65 22 3A 22 42 4C 49 4E 44 5F 42 4F 58 22 7D 2C 22 75 72 6C 22 3A 22 61 73 73 65 74 73 2F 69 63 6F 6E 73 2F 6C 61 62 2F 77 65 61 70 6F 6E 5F 63 6F 6D 70 6F 6E 65 6E 74 73 2F 6F 70 74 69 63 61 6C 5F 73 63 6F 70 65 2E 70 6E 67 22 7D 5D 2C 22 5F 63 6D 64 22 3A 22 67 65 74 5F 70 6C 61 79 65 72 5F 6C 61 62 22 2C 22 63 6F 6D 70 6F 6E 65 6E 74 5F 6D 61 78 5F 74 69 6D 65 22 3A 38 36 34 30 30 2C 22 73 75 63 63 65 73 73 22 3A 30 7D 7D 7D 00)
[java] at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:165)
[java] at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
[java] at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
[java] at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
[java] at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:220)
[java] at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:264)
[java] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
[java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[java] at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
[java] at java.lang.Thread.run(Thread.java:619)
[java] Caused by: org.apache.mina.common.BufferDataException: Line is too long: 49497
[java] at org.apache.mina.filter.codec.textline.TextLineDecoder.decodeNormal(TextLineDecoder.java:247)
[java] at org.apache.mina.filter.codec.textline.TextLineDecoder.decode(TextLineDecoder.java:114)
[java] at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:158)
[java] ... 9 more

[Lapo]

Hi,
the socket classes that handle the data traffic in the Java API have a limit of 32KB per message. Normally this is more than sufficient for any application. Unfortunately there is no way from code to change these limits because they are set when the protocol encoder/decoder are created and can't be changed at runtime.

The only way is that we recompile one of the API libraries with a higher limit (let's say 128-256KB or so)

Another way around this is to understand why so much data is transferred in one shot and see if you can reduce it. Maybe this is a very long room list? Or a very large extension request? In the latter case you might switch to a more compact protocol such as JSON if you haven't already done it.

[dromos]

Hi,
the socket classes that handle the data traffic in the Java API have a limit of 32KB per message. Normally this is more than sufficient for any application. Unfortunately there is no way from code to change these limits because they are set when the protocol encoder/decoder are created and can't be changed at runtime.

The only way is that we recompile one of the API libraries with a higher limit (let's say 128-256KB or so)

Another way around this is to understand why so much data is transferred in one shot and see if you can reduce it. Maybe this is a very long room list? Or a very large extension request? In the latter case you might switch to a more compact protocol such as JSON if you haven't already done it.

This is an extension response and we are already using JSON. The Flash client API is handling the message just fine, so this was never an issue until I tried to write a Java-based load tester. The messages are so big because we front-load a lot of data.

[EDIT] I may have misunderstood. Are you suggesting you could recompile the client library to support an increased buffer size? Is this a possibility? Perhaps you could let us override the default through a property passed to the JVM? We are planning on building a lot of games on SFS, and being able to load test them using the Java client API is really important for us.

Thanks.

[Lapo]

[EDIT] I may have misunderstood. Are you suggesting you could recompile the client library to support an increased buffer size? Is this a possibility? Perhaps you could let us override the default through a property passed to the JVM? We are planning on building a lot of games on SFS, and being able to load test them using the Java client API is really important for us.

Yes that's correct.
We are already working on it. If all goes well we'll upload an update by the end of the week. Stay tuned

[Lapo]

One more thing: please send us an email (click Support > Contact Us in the main menu) and we'll provide an early release that you can test.

Thanks

[dromos]

One more thing: please send us an email (click Support > Contact Us in the main menu) and we'll provide an early release that you can test.

Thanks

That's great, thanks! I have sent the support e-mail as requested.

[jsanmarco]

Hi,
has this been solved? we are getting the same error.

Regards,
Juan

[Lapo]

Yes the problem was solved with the latest Java API version released several months ago.
We added two external settings that can be passed to the JVM to specify the message buffer size (incoming/outgoing).

Code: Select all

sfs.client.inDataMaxSize=XXX
sfs.client.outDataMaxSize=XXX

Example:

Code: Select all

java -jar <yourapp> -Dsfs.client.inDataMaxSize=64000 -Dsfs.client.outDataMaxSize=64000

The default is 32000 (32Kb)

Download: http://smartfoxserver.com/labs/API/