Unreliable message latency

[preecep]

I have spent 2 days trying to fix (or solve) a latency problem with SmartFox. My setup is:

Client Side: Flash CS3, Firefox, IE, SF API 1.6.2
Server Side: Windows 2003 with SFS 1.6.2, Centos 5 with SFS 1.6.2
Server Script: ActionScript, Java

Server is on a 100Mbit connection so is the client using. Port 9338.

I have created a flash widget that connects to a room (with an extension) and simple does one of the following:

* Uses roundTripBench() and prints it out.

* creates a time stamped object and uses sendObjectToGroup() to send the object back to the same user. It then prints out the time the message has taken to do that trip. (ie a ping)

* Creates a time stamped object and uses sendXtMessage() to send the object back via the extension to the same user. It then prints out the time the message has taken to that trip (ie a ping)

The actual ping to the server is 100ms +/- 5.

So why do we get the following round trip times:

handleRoundTripResponse:433
handleRoundTripResponse:519
handleRoundTripResponse:390
handleRoundTripResponse:471
handleRoundTripResponse:353
handleRoundTripResponse:339
handleRoundTripResponse:8
handleRoundTripResponse:125
handleRoundTripResponse:392
handleRoundTripResponse:380
handleRoundTripResponse:360
handleRoundTripResponse:462

I have tried every combination of Server OS / Extension type / Client PC / Browser / ping technique I can and the results are always the same.

As far as I am concerned this makes SFS worse than using HTTP, so what is happening? I have spent (yet another) 2 days trying to fix a problem with SFS which simply should not be happening. If I cannot fix this issue I think we will have to return the software as 'not fit for purpose'.

What tests can I run to locate the problem?

[preecep]

After more investigation this morning we are getting a 200ms lag to other PC's in our office running SFS. So it may be an internal problem of some kind.

[potmo]

if you runs smartfox on localhost you get pingtimes around 0 to 2 ms.
I think its your router thats the thief in the drama.

[preecep]

We cut out the router and just used static IP's and our switch. Still had the same results.

Internal tests result from running client fla inside CS3:

XP Client -> Local host = 3ms
XP Client -> Centos = 50 +/- 5
XP Client -> XP = 200 +/- 50
XP Client -> Vista = 200 +/- 5

All normal pings locally are 0 ms.

I have just tried it from one of our hosted servers to another (using a web browser to run the swf) and that too returns 150ms ping when normal ping is 40ms.

I am at a loss to explain what is causing this problem as nothing is common to ALL tests except our fla code and that doesn't have a problem when connecting to a SmartFox server on local host...

[preecep]

I have rebuilt some of the code from our connection fla in a new fla and that works fine, in line with normal ping. More investigation...

[preecep]

I have found the problem.

In our code we were sending an object to the players in the room in a function that was called using an interval:

Code: Select all

smartfox.sendObject(o);

followed immediately by:

Code: Select all

smartfox.sendObjectToGroup(o,[userid]);

or a:

Code: Select all

smartfox.roundTripBench();

or a:

Code: Select all

smartfox.sendXtMessage(xtName,cmdname,o,"xml");

In all 3 cases, we receive ping times that are very bad and fluctuate.

If we comment out the first SendObject() that lag goes away.

So, is this normal behaviour?? Can SmartFox only send one message per frame?

[Lapo]

So, is this normal behaviour?? Can SmartFox only send one message per frame?

No, there's no such limit.

I tested a similar scenario locally and didn't find anything strange.
Whether I send only a roundTripBench or I also use a sendObject and a sendPublicMessage (in the same frame) I get a consistent ping-pong time of 8-10ms.

With this said, if you find yourself sending multiple requests on each frame you should probably optimize it to a single call, when possible.
In your example you send an object from client side and you also invoke an extension which I expect to do some other work on the server side.

In this case you could optimize it by sending the object to your extension which in turn will perform the server side actions and broadcast the object back to its recipients.

[preecep]

I will create a fla to demonstrate this problem.

Please make sure you test it against a server on the internet, as local host does not seem to cause this problem.

Also, the first message goes through ok (if it was the ping) but it is the second message that lags.

I did some further testing sending many messages at the same time (to the server and from the server) and it appears to cause problems in both directions. ie the messages arrive spread out over 200-300 ms.

[preecep]

This problem can be easily seen by using this code:

Code: Select all

import it.gotoandplay.smartfoxserver.*;

var smartfox:SmartFoxClient;
var _tick = 0;
var _state=1;
var _ip="65.98.62.162";
var _port=9339;

InitSmartFox();

smartfox.connect(_ip,_port);

function Complete(tf) {
   trace("Complete "+_state+":"+tf);
   switch(_state++) {
      case 1:
         Login();
         break;
      case 2:
         AutoJoin();
         break;
      case 3:
         _root.onEnterFrame = function() {
            // every 1/2 sec do pings
            if (++_root._tick == 20) {
               // send dummy object
               var o:Object={action:"dummy"};
               smartfox.sendObject(o); // comment out to return pings to normal
               
               // then perform a smartfox ping
               smartfox.roundTripBench();
               // or a SendObject ping
               //smartfox.sendObjectToGroup({action:"ping",d:getTimer()},[smartfox.myUserId]);
               // or a PublicMessage ping
               //smartfox.sendPublicMessage(""+getTimer());
               _root._tick = 0;
            }
         }
         break;
   }
}
function InitSmartFox() {
   smartfox = new SmartFoxClient();
   smartfox.onConnection = handleConnection;
   smartfox.onLogin = handleLogin;
   smartfox.onJoinRoom = handleRoomJoin;
   smartfox.onObjectReceived = handleObjectReceived;
   smartfox.onPublicMessage = handlePublicMessage;
   smartfox.onRoundTripResponse = handleRoundTripResponse;
   smartfox.onExtensionResponse = handleXTResponse;
   smartfox.debug=false;
   smartfox.smartConnect = false;
}
function handleConnection(success:Boolean) {
   trace("handleConnection:"+success);
   _root.Complete(success);
}
function handleRoundTripResponse(t:Number) {
   trace("RoundTripResponse:"+t);
}
function handleObjectReceived(o:Object, sender:User) {
   trace("Object Latency:"+(getTimer()-int(o.d)));
}

function handlePublicMessage(msg:String, sender:User) {
   trace("Message Latency:"+(getTimer()-int(msg)));
}
function Login() {
   smartfox.login("sftris","testplayer");
}
function Connect() {
   smartfox.connect(ip,port);
}
function AutoJoin() {
   smartfox.autoJoin();
}
function handleLogin(resObj:Object) {
   if (resObj.success) {
      _root.Complete(true);
   } else {
      _root.Complete(false);
   }
}
function handleRoomJoin(roomObj:Room) {
   trace("Successfully joined room: " + roomObj.getName())
   _root.Complete(true);
}


It will connect to our smartfox server which is a completely default install of 1.6.2. If you leave in the first dummy SendObject the roundtrip time will be very high. If you comment it out, you should see it return to normal.

[Lapo]

Thanks for the code. I've performed 4 different tests (each one 3 times) and didn't find anything strange.

Test 1 (remote)
1) ping <yourServerIP> = 142 - 143ms.
2) Launch your flash test with sendObject:

Code: Select all

handleConnection:true
Complete 1:true
Complete 2:true
Successfully joined room: undefined
Complete 3:true
RoundTripResponse:150
RoundTripResponse:150
RoundTripResponse:150
RoundTripResponse:149
RoundTripResponse:150
RoundTripResponse:149
RoundTripResponse:149
RoundTripResponse:150
RoundTripResponse:149
RoundTripResponse:150
RoundTripResponse:150
RoundTripResponse:149
RoundTripResponse:150
RoundTripResponse:149

Test 2 (remote)
1) ping <yourServerIP> = 142 - 143ms.
2) Launch your flash test without sendObject:

Code: Select all

handleConnection:true
Complete 1:true
Complete 2:true
Successfully joined room: undefined
Complete 3:true
RoundTripResponse:151
RoundTripResponse:159
RoundTripResponse:150
RoundTripResponse:151
RoundTripResponse:151
RoundTripResponse:151
RoundTripResponse:151
RoundTripResponse:150
RoundTripResponse:151
RoundTripResponse:151
RoundTripResponse:150
RoundTripResponse:151

Test 3 (local)
1) ping <ourInternalIP> = 2 - 3ms.
2) Launch your flash test with sendObject:

Code: Select all

handleConnection:true
Complete 1:true
Complete 2:true
Successfully joined room: undefined
Complete 3:true
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8
RoundTripResponse:8

Test 4 (local)
1) ping <ourInternalIP> = 2 - 3ms.
2) Launch your flash test with sendObject:

Code: Select all

handleConnection:true
Complete 1:true
Complete 2:true
Successfully joined room: undefined
Complete 3:true
RoundTripResponse:9
RoundTripResponse:9
RoundTripResponse:9
RoundTripResponse:8
RoundTripResponse:9
RoundTripResponse:9
RoundTripResponse:10
RoundTripResponse:9
RoundTripResponse:9
RoundTripResponse:8

As far as I can see there's no significant difference between the test with or without the sendObject call. Most of the relevant lag time is due to the network access.

The local test was run with a Linux box running a clean SFS 1.6.2 installation. CPU = P4 3Ghz, OS = Ubuntu Linux 7

[preecep]

Strange.

I have uploaded a test to:

http://www.casualcollective.com/pingtest3.swf

You can toggle the extra SendObject on and off (starts on).

On all our machines here and via a separate server, we get the same bad pings.

[Lapo]

Doesn't work for me. It says:

handleConnection:true
Complete 1:true
Complete 2:false

[preecep]

Some one else is keeping that user logged in I bet, so it fails to login when you do it.

I have uploaded a new version which uses a random user name. You may need to clear your web cache.

[Lapo]

Thank you. I see the differences if I use a browser, but I don't if run the swf in the standalone player.
I'd need to investigate a little more. Can you send me the FLA? Just zip it and send it to our email address.

Thanks

[potmo]

can it be a slow DNS lookup if youre using urls instead of IP???