Load Testing

[wiggly]

Hello,

I am currently trying to load test an application that is provided by a java extension.

The application works and is functionally complete, we're just trying to see how the thing handles load.

I have a load test client that basically simulates N clients that connect with some average arrival rate.

My current problem is that if I set the arrival time to a smaller value then lots of clients cannot connect. The following is an excerpt from the load test logfile.

Before this other clients are working away, then there is the "Invalid connection = null" line on its own, which must be coming from a print statement in one of the smartfox libraries, or its dependencies.

Once this happens you can see the call trace (below) that then gets thrown when disconnect is called. The root cause of this I beleive is that the XMLSocket code isn't actually able to connect to the default 9339 port for some reason. Whether this is the client code or the server code's fault I'm not sure.

I have already tried upping the number of handlers available on the server side and playing with the config values for a while, I was wondering if anyone has had success getting a load test running through the Java client code?

For reference I have tried configuring the load tester to fire off a new client from 250ms-750ms right up to 1000ms-3000ms, and even with an average arrival rate of 2 seconds I still get these errors.

Am I expecting too much? Personally I think 750ms is a lifetime...

I will be looking at this more today and playing around. What I would find incredibly interesting is the source for the XMLSocket code so I could actually run the thing under a debugger and find out what was really going on here.....

Oh, I have an unlimited license by the way, if I need to dig out any details for support just let me know.

-----
Invalid connection = nullM
2009-04-09 10:45:02 DEBUG pool-1-thread-3 TrappedClient - handleEvent : onConnectionM
2009-04-09 10:45:02 DEBUG pool-1-thread-3 TrappedClient - handleConnectionM
2009-04-09 10:45:02 WARN pool-1-thread-3 TrappedClient - connection failedM
2009-04-09 10:45:02 ERROR pool-1-thread-3 TrappedClient - handle event caught exceptionM
java.lang.NullPointerExceptionM
at it.gotoandplay.utils.net.xmlsocket.XMLSocket.close(XMLSocket.java:152)M
at it.gotoandplay.smartfoxclient.SmartFoxClient.disconnect(SmartFoxClient.java:1266)M
at uk.co.bbc.rad.trapped.client.TrappedClient.disconnect(TrappedClient.java:278)M
at uk.co.bbc.rad.trapped.client.TrappedClient.handleConnection(TrappedClient.java:368)M
at uk.co.bbc.rad.trapped.client.TrappedClient.handleEvent(TrappedClient.java:325)M
at it.gotoandplay.smartfoxclient.SFSEventDispatcher.dispatchEvent(SFSEventDispatcher.java:96)M
at it.gotoandplay.smartfoxclient.SmartFoxClient.dispatchConnectionError(SmartFoxClient.java:4550)M
at it.gotoandplay.smartfoxclient.SmartFoxClient.tryBlueBoxConnection(SmartFoxClient.java:4502)M
at it.gotoandplay.smartfoxclient.SmartFoxClient.onError(SmartFoxClient.java:4471)M
at it.gotoandplay.utils.net.xmlsocket.XMLSocket.fireErrorEvent(XMLSocket.java:206)M
at it.gotoandplay.utils.net.xmlsocket.XMLSocket.send(XMLSocket.java:178)M
at it.gotoandplay.smartfoxclient.SmartFoxClient.send(SmartFoxClient.java:4149)M
at it.gotoandplay.smartfoxclient.SmartFoxClient.onConnect(SmartFoxClient.java:4450)M
at it.gotoandplay.utils.net.xmlsocket.XMLSocket.fireConnectEvent(XMLSocket.java:191)M
at it.gotoandplay.utils.net.xmlsocket.XMLSocketClientHandler.onConnect(XMLSocketClientHandler.java:30)M
at org.xsocket.stream.NonBlockingConnection.onConnectEvent(NonBlockingConnection.java:823)M
at org.xsocket.stream.Connection$HandlerCallback.onConnect(Connection.java:1584)M
at org.xsocket.stream.io.impl.IoMultithreadedHandler$IOEventHandler$1.run(IoMultithreadedHandler.java:142)M
at org.xsocket.stream.io.impl.IoMultithreadedHandler$TaskQueueProcessor.processTask(IoMultithreadedHandler.java:277)M
at org.xsocket.stream.io.impl.IoMultithreadedHandler$TaskQueueProcessor.run(IoMultithreadedHandler.java:265)M
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)M
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:90
-----

[Lapo]

Hi, it might depend on the underlying xsocket library which we have found to have some problems handling very fast connections.
Because of this we have created a parallel version of the same API which bases the socket connections on Apache MINA. This also allows the API to be compatible with Google's Android, which is a nice plus

You can download the MINA-based API here.
Let us know if it solves the issue.

[wiggly]

Hi, it might depend on the underlying xsocket library which we have found to have some problems handling very fast connections.
Because of this we have created a parallel version of the same API which bases the socket connections on Apache MINA. This also allows the API to be compatible with Google's Android, which is a nice plus

Thanks for the quick reply, I will give this a go when I get a chance, I've just started breaking a load of things to refactor some stuff.

One thing to note, I think it *is* the xmlsocket library. It doesn't seem to be the server since I can get a simple perl script to connect up as fast as one new connection every 64ms......until I run out of descriptors....

Thanks again,

Nige

[Lapo]

One thing to note, I think it *is* the xmlsocket library.

Yep, and I agree with you. In fact the XMLSocket library actually uses xsocket for the socket-level operations and I think that the MINA-based version will behave differently (and hopefully better )

[wiggly]

Hi there,

I tried that new set of libraries this morning and unfortunately the results are mixed.

Yes, they do not fail whilst trying to connect at between 250-750ms, haven't tried any faster yet.

The problem is that at about 160 concurrent connections I get a "too many open files" error, which is weird since I would expect to be able to go to about 1024. I guess maybe the libraries are using a number of filehandles for things other than networking, but 160 out of 1024 sounds low.

I will see if I can bump this number up and try again, but for right now I have no good way of load testing. Can you let me know how many concurrent connections you can get using these libraries?

[Lapo]

Keep in mind that the client API don't use advanced non-blocking/single threaded strategies.
Usually a client connection requires two threads (MINA does that, and I think xsocket does that as well)

If you want to heavily bombard the server with requests you might need a non-blocking custom client library that works at the protocol level (in other words it sends raw data)

p.s. = as far as I know one client socket is opened for each connection and no other file-handles should be used (I can't imagine what they would be used for)

[wiggly]

Keep in mind that the client API don't use advanced non-blocking/single threaded strategies.
Usually a client connection requires two threads (MINA does that, and I think xsocket does that as well)

This seems to be beside the point, I'm not having a problem with throughput, I'm hitting some kind of hard ceiling on the number of connections the Android/MINA libs will create, and that being far lower than I would expect.

If you want to heavily bombard the server with requests you might need a non-blocking custom client library that works at the protocol level (in other words it sends raw data)

I can't say I have time to go and write something that talks the protocol from scratch....that was the whole point of trying to use the client libraries

p.s. = as far as I know one client socket is opened for each connection and no other file-handles should be used (I can't imagine what they would be used for)

Yes, it doesn't make much sense to me either, anyway I'll let you know what I find when I get a chance to have another look.