High latency when playing a game

Post here your questions about the HTML5 / JavaScript for SFS2X

Moderators: Lapo, Bax

jdv145
Posts: 21
Joined: 11 May 2013, 13:13

High latency when playing a game

Postby jdv145 » 13 Nov 2013, 15:35

I'm having a problem with the latency going through the going through the roof when playing an 1v1 game.

First my configuration:

I'm using two independent chrome browsers to connect to smartfox, which is run on an other computer inside my LAN. Normally, during non-gameplay, the latency is between 1-2ms (according to the inbuild PING-PONG measurements).

Client: i7-2600 with 12GB ram and win7 x64
Server: i3-3110 with 16GB ram and win7 x64
Both connected using gbit having a very low ping (when using the dos ping command). The CPU load is always low (0-10%) on both computers and there is plenty of free RAM.

As soon as i start the game and don't control anything, the latency usually stays between 2-3ms, (but sometimes it starts at 100ms+ immediately).

If i start controlling the game, the latency goes up significantly and often around 250 ms, but not always.

The gamestate is a string of 400 length which gets transmitted (using a roomvariable) at each keypress or after each time-tick (when the game changes), which is about every second. During active gameplay there are +/- 3 update's (3 keypresses) a second .

I have NOT spend that much time on it to trace the cause, but so far i consistently get very high ping times sooner or later when playing a game.

Is my usecase to much of a load for smartfox (which i can hardly imagine)? Is the measured time incorrect because javascript is not multithreaded and somehow this mangles the times? What can i best to to find the cause?
User avatar
rjgtav
Posts: 2813
Joined: 19 Apr 2009, 11:31
Location: Lisbon, Portugal

Re: High latency when playing a game

Postby rjgtav » 14 Nov 2013, 01:51

Hi,

One thing that directly came to my mind, can you check if the browser is firing multiple times the keypress event when a key is down, instead of firing it ony once each time you press it?
I'd also suggest you to check the system queues and the server's load.

Cheers
Skills: SFS Pro, SFS2X, AS2.0/AS3.0, Java, HTML5/CSS3/JS, C#
Portfolio: https://rjgtav.wordpress.com/
SFS Tutorials: http://sfs-tutor.blogspot.com/ - Discontinued. Some examples may be bugged.
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 14 Nov 2013, 18:51

rjgtav wrote:Hi,

One thing that directly came to my mind, can you check if the browser is firing multiple times the keypress event when a key is down, instead of firing it ony once each time you press it?
I'd also suggest you to check the system queues and the server's load.

Cheers


I was not detailed enough with my previous post, my humble apologies.

I did check the admin panel of smartfox to see if anything is out of the ordinary (and i expected something to be wrong with these high latencies), but nothing jumped out (so also no long queue).
I did not check if multiple keystroke's were fired, but i did check when an update was send and received and the frequency was as it should be (1 update per key or 1 per sec when doing nothing).

I am more then willing to do some more investigation, but i i don't exactly know where to look to find the bottleneck.

Regards,

jan

edit
I played again a couple games, this time from a laptop on wifi and my workstation. Just to see what happens i tried to keep a key down on *both* clients and to press alternating left&right as fast as i could.

This results in a network traffic of max 32KB/s peak in the admin panel (Global server status tab) and always queues of 0% (System queue status tab)

This time one client stayed arround 2ms latency and the other 70-100ms.

edit2
using firefox (instead of chrome) on both the workstation and laptop gives me a latency of 40ms and 18ms
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 15 Nov 2013, 11:45

I'm not sure if i should add a 3rd edit or add a new post.... but since there is some new stuff i tried a guess a new post is in place.

I couldn't make much sense of the jumpiness of the inbuild ping-pong so i added a latency meter myself.

Actually i did it twice, one latency timer sends a message to the server which will send it to an other player and then back again. So this one has 4 hops. This one uses private-chat-messages (using the data object).

I also added a latency timer which goes to the server using an extension-request and then smartfox immediately sends it back to the sender (2 hops).

The interval is 1 second and i looked at the average of 5 measurements and the last measurement.

I noticed the latency-times differs very much from the inbuild ping-pong. i mostly watched the 2 hops timer and usually the ping is around 1 ms, sometimes a little bit lower. As soon as i play a game (or if i send 2 ping messages almost simultaneously) the ping increases a lot (+/- 100ms ). I also experimented with adding a string of 400 characters and increasing the frequency. The ping times are very low but as soon as you start send even a little traffic this sometimes interferes and causes massive latency.

During all testing i also did a regular ping from a dos box from the workstation to the computer running sfs... this was without exception <1ms

I also looked at potential places in my game, which might take a long time and maybe can cause a delay. I even disabled the drawing algorithm. So far i could not find any places where the javascript 'hangs' a significant amount of time. Everything is nice and snappy except the latency :cry:

I tested everything with one firefox and one chrome to see if this makes a difference (there was a small, i think statistically significant difference, but it did not explain the huge latencies).

So i'm still in search for getting acceptable latencies...
User avatar
Bax
Site Admin
Posts: 4522
Joined: 29 Mar 2005, 09:50
Location: Italy
Contact:

Re: High latency when playing a game

Postby Bax » 15 Nov 2013, 17:44

It is quite difficult to understand what is going on.
Maybe you can send an example with minimal code showing this behavior?
Also, it is not clear if you are experiencing actual issues in your game or your only hint about this lag is the measurement you get.
Paolo Bax
The SmartFoxServer Team
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 15 Nov 2013, 21:40

The basic problem is that with a minimal workload the latency is far to high. I don't experience any problems now (the game i'm playing is tetris), but with the delays i'm seeing i am sure this will be problematic for some games that will be created in the future (this is also one reason i made my own pinger so i know exactly what gets measured ).

But testing with a whole game with the in build latency measurements is difficult to debug, so i decided to make my own ping functionality and see how if i could get better ping times this way. You can see it below.

Basically There are 2 different custom ping-meters. If there are two clients in a room, the 'PersonalMsgPing' measures the time between client1 -> server -> client2 -> server -> client1 (4 hops). As the name implies it uses personal-message-requests.

Since this resulted in high latencies i decided to make an even simpler version which uses extension-requests which goes like this:
client1 -> server -> client1

The code for the smartfox extension is below:

Code: Select all

    public class CustomPingHandler extends BaseClientRequestHandler {
    @Override
    public void handleClientRequest(User sender, ISFSObject params) {
        getParentExtension().send(CustomKeys.CUSTOM_PONG, params, sender);
    }   
}


The javascript which measures the time is below. I know this is not the best code, and i slightly abuse the personal messages, but it will be removed once the latency is okay. It's just to have some extra values besides the build-in ping-pong measurements.

Code: Select all

PING_MANAGER = function(sfs, name) {
    this.sfs = sfs;
    this.name = name;

    this.usePersonalMsgPing = true;
    this.useCustomPing = true;

    this.interval = 1000;
    this.numHistory = 5;

    this.customPings = [];
    this.pmPings = {};

    this.avg = null;


    if (this.usePersonalMsgPing) {
        UT.setIntervalWithContext(this.sendPersonalMsgPing, this.interval, this);
    }

    if (this.useCustomPing) {
        UT.setIntervalWithContext(this.sendCustomPing, this.interval, this);
    }
};

//<editor-fold defaultstate="collapsed" desc="PING_MANAGER functions">
PING_MANAGER.prototype.addPersonalMsgPing = function(name, ping) {
    UT.debug('Latency to ' + name + ': ' + ping + " ms");
    var arr = this.pmPings[name];
    if (!arr) {
        arr = [];
        this.pmPings[name] = arr;
    }
    this.pmPings[name] = this.addPing(arr, ping);
};

PING_MANAGER.prototype.addCustomPing = function(ping) {
    var arr = this.customPings;
    this.customPings = this.addPing(arr, ping);

    this.avg = arr.average();
};

PING_MANAGER.prototype.addPing = function(arr, ping) {
    if (arr.length > this.numHistory) {
        arr = arr.slice(1);
    }
    arr.push(ping);
    return arr;
};

PING_MANAGER.prototype.sendPersonalMsgPing = function() {
    if (!this.sfs || !this.sfs.isConnected()) {
        return;
    }
    var room = this.sfs.roomManager.getRoomByName(CFG.sfsConfig.lobbyRoom);
    if (room == null || !room.isJoined) {
        return;
    }

    var users = room.getUserList();
    for (var i = 0; i < users.length; i++) {
        var u = users[i];
        if (u.name == this.name) {
            continue;
        }
        var obj = {
            noChat: true,
            cmd: CFG.customKeys.PERSONAL_MSG_PING,
            time: new Date().getTime()
        };
        var msg = 'Ping to: ' + u.name;
        var id = u.id;
        //UT.log('sending ping to ' + this.name);
        //UT.setTimeoutWithContext(function() {
        UT.sendPrivateMessageRequest(this.sfs, msg, id, obj);
        //}, 1, this);
    }
};

PING_MANAGER.prototype.sendCustomPing = function() {
    var room = this.sfs.roomManager.getRoomByName(CFG.sfsConfig.lobbyRoom);
    if (room == null || !room.isJoined) {
        return;
    }
    var obj = {time: new Date().getTime()};
    UT.sendExtensionRequest(this.sfs, CFG.customKeys.CUSTOM_PING, obj);
};

PING_MANAGER.prototype.onPrivateMessage = function(sender, msg, data) {
    if (!data) {
        return;
    }
    if (data.cmd == CFG.customKeys.PERSONAL_MSG_PING) {
        if (sender.name != this.name) {
            var msg = 'Pong from: ' + this.name;
            data.cmd = CFG.customKeys.PERSONAL_MSG_PONG;
            UT.sendPrivateMessageRequest(this.sfs, msg, sender.id, data);
        }
    } else if (data.cmd == CFG.customKeys.PERSONAL_MSG_PONG) {
        if (sender.name != this.name) {
            var delta = new Date().getTime() - data.time;
            delta = delta / 4;
            this.addPersonalMsgPing(sender.name, delta);
        }
    }

};

PING_MANAGER.prototype.onExtensionResponse = function(cmd, params, sourceRoomId) {
    if (cmd == CFG.customKeys.CUSTOM_PONG) {
        var delta = new Date().getTime() - params.time;
        delta = delta / 2;
        this.addCustomPing(delta);
    }
};


Even without playing a game and having only traffic coming from the ping measurements, the latencies are all over the place. Having decent hardware connected to gbit, this is somewhat of an unwelcome surprise knowing that smartfox is designed for much much more load then this.
User avatar
rjgtav
Posts: 2813
Joined: 19 Apr 2009, 11:31
Location: Lisbon, Portugal

Re: High latency when playing a game

Postby rjgtav » 16 Nov 2013, 02:40

Hi.

Can you test the lag results with all the major browsers? Chrome, Firefox, IE 10 or 11, Opera and maybe Safari? You mentioned that the lag changed a lot from a browser to another, so maybe there's some clue there.
In order to get the best results, use 1 browser at a time, with only your game opened and make sure that there's not another software that is using up the network resources.
And try also testing by running the app and the server on the same machine. On this scenario, you should get 0ms of delay, otherwise, something is slowing down your game's execution.

Oh, and use only the ExtensionRequest method, as the other one using private messages is much slower ;)

Thanks
Skills: SFS Pro, SFS2X, AS2.0/AS3.0, Java, HTML5/CSS3/JS, C#
Portfolio: https://rjgtav.wordpress.com/
SFS Tutorials: http://sfs-tutor.blogspot.com/ - Discontinued. Some examples may be bugged.
User avatar
Bax
Site Admin
Posts: 4522
Joined: 29 Mar 2005, 09:50
Location: Italy
Contact:

Re: High latency when playing a game

Postby Bax » 16 Nov 2013, 14:54

I'm not sure it is the right approach to concentrate your attention on the numbers you get from the ping.
Anyway it would be interesting to see if you get similar results using a different technology, like Flash for example.
Paolo Bax
The SmartFoxServer Team
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 16 Nov 2013, 16:56

rjgtav wrote:Hi.

Can you test the lag results with all the major browsers? Chrome, Firefox, IE 10 or 11, Opera and maybe Safari?
Thanks


So far i have tested the newest chrome and firefox and i just used IE10. Although i think there is a discernible difference, its not much more then say 10%. This can be explained by the performance of the javascript-engine used by the browser. As far as i know this can not be the cause of the heavy latencies i have seen so far.

Testing with a single IE10 client on the server gives me a ping of 1-2ms. If i also connect using my laptop (quadcore i7-2720, IE10) both clients jump between 1-100ms.

I thought maybe it is a structural thing in the websocket-standard, but using google i could not find any indication this is the case and i could not find anybody who also had a problem achieving an acceptable latency using websockets.

Bax wrote:I'm not sure it is the right approach to concentrate your attention on the numbers you get from the ping.
Anyway it would be interesting to see if you get similar results using a different technology, like Flash for example.


I noticed the big latencies before and i thought this will we solved in a later stage of the development. But for some time i have seen it and i am pretty sure that if i can't even send the simplest and shortest message using ExtensionRequest its impossible to make a decent gameserver for my customer.

I have considered using flash instead of websockets, but so far i thought other test's might bring me faster to the solution of this problem. If it works correctly in flash it proves that the websocket implementation might indeed be faulty, but i'm not much closer to finding the solution for websockets. If flash is OK, then i know it is specifically in the websockets, but i already know i have a problem using websockets. My guess is that the flash implementation will work since it is a much more mature part of smartfox and if there were bugs in it, my guess is that it would have surfaced long before i started using smartfox by somebody else.

I think the best thing now is to take an html5 example from the site and adept it to show the latency. Then i have a clear reproducible case which shows how fast or slow smartfox is. If i did anything wrong, then it will show up quickly and i can pinpoint it. If it doesn't perform well... then the ppl behind smartfox have an example to work with and investigate it farther if they would like to.

Doesn't (the company behind) smartfox do their own benchmarks every release to prevent performance degradation? What kind of numbers do you get? Can i maybe get an example to test on my computers for comparison ?

PS i already spend a couple of afternoons on this problem, but it's hard to clearly explain everything i've done (and not that useful if it was a dead end). I sure hope i don't have to test the internals of smartfox :shock:

PPS if it was my own server, i would measure @server-side the time between an incoming and an outgoing response. It would be even better if this would be visible one way or the other in the admin panel. This, in the end, is how fast smartfox is perceived to be performing from the outside. I know not every incoming request results in an outgoing response, but it could be done for i.e. only the internal PING-PONG.

PPPS maybe i'l try to use a packet sniffer. If i'm lucky i should be able to exactly see where the bottleneck is. It has been some time i use one though :o
User avatar
Lapo
Site Admin
Posts: 21680
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High latency when playing a game

Postby Lapo » 16 Nov 2013, 19:10

The fastest way for us to verify a possible problem is to see a reproducible case.
If you can please isolate the code that causes the issue in a simple client example and send it to us, we will take a look and get back to you.

You can zip the material and send it our support@... email box with a reference to this thread.

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 16 Nov 2013, 21:14

I might have found the issue, but i'm very much interested about what you guys think. Although the problem is not yet solved, i'm a little bit optimistic about my findings.

I used wireshark to monitor the computer on which smartfox is running. Im not a pro in packetsniffing (not by a long shot), so it took me some time to find my way around and not get overwhelmed by the large amount of packets.

What i did was very simple... i just ran my custom-ping where each message/ping got it's own index. If the latency was higher then 80ms i would print the index from javascript. This way i knew which ping was slow and which one was fast by looking at the index.

Using the indices, i could find every packet in the logs of wireshark and determine if it was a slow or fast one and compare them.

I found out that the fast packets were in a single frame, but the slower ones were combined with other data. I remembered from a long time ago when developing my own socket-server that there is a settings which combines traffic to increase the performance, and after some searching i found out it is tcpnodelay. To increase the network efficiency, packets can be delayed so other packets with the same destination can be merged. Something like combining two letters into a single envelope.

Something like this would seem very logical to me, because there is hardly any RAM/CPU/network load. Also the latency doesn't change if i add/remove 400 characters of data or increase the ping-frequency. So it doesn't look like a typical congestion problem.

I'm not sure why, but if there is 1 client connected, messages don't get combined, but if there are 2 clients connected, messages often do get combined and the latency shoots up. If i could just prevent this from happening, i think i'm okay. I'm not sure if this is really tcpnodelay in action or something smartfox does to increase the network-efficiency but it does look like something is doing this.

I have stored the wireshark logs (2 sessions) for further analysis which contain some fast and slow ping-messages.

NB i was only able to monitor the 'pong' coming from the server, not the 'ping' going to the server. Wireshark shows the content of the packets (as hex and text) but i could not find any packets with with 'ping' as plain text, so i think the content of some packages are zipped (or otherwise compressed) or something.

PS i did some steps as show in http://www.em-zero.com/index.php/2009/12/tcpackfrequency-tcpnodelay-and-more/ to disabled tcpnodelay but it didn't help. I also looked at core.xml of smartfox to confirm it is really turned of.

Lapo wrote:The fastest way for us to verify a possible problem is to see a reproducible case.
If you can please isolate the code that causes the issue in a simple client example and send it to us, we will take a look and get back to you.

You can zip the material and send it our support@... email box with a reference to this thread.

Thanks


Thanks, i think if my observation above doesn't bring me/us closer to a solution, i will try one of the html5 examples and if i get the same problem ill send it to you guys.

edit: Because of my previous observation i decided to test my network using a seperate socketserver i quickly put together using node.js. This gives very nice consistent latencies between 1-2 ms on all browsers. I also tested the server on my smartphone over the internet (not LAN) and even then i get pretty decent timings.
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 17 Nov 2013, 15:48

A new post since the other one gets to large otherwise...

I've created a 3rd socket-server in dart for comparison. So now i have 3 running on the server,
- smartfox : latency is jumping
- one made in node.js : latency always low
- one made with dart.: latency always high

From what i have seen in the code and documents i think the one in node.js is explicetly using tcpnodelay

Code: Select all

        // The Nagle Algorithm makes more efficient use of network resources
        // by introducing a small delay before sending small packets so that
        // multiple messages can be batched together before going onto the
        // wire.  This however comes at the cost of latency, so the default
        // is to disable it.  If you don't need low latency and are streaming
        // lots of small messages, you can change this to 'false'
        disableNagleAlgorithm: true,


Dart is explicetly disabling tcpnodelay:

Code: Select all

TCP_NODELAY is disabled by default.


I also played around with the registry settings in my workstation and this does indeed help to get consistent low latencies... but i can't ask all players on the server to use regedit of course. On my server i did set TcpAckFrequency and TCPNoDelay. For now my assumption is that the cause is the nagle algorithm. This can be solved by setting this on the client (which is not an acceptable solution) but *also* on the server (since my node.js server is running with low latencies no mater what the TCPNoDelay/TcpAckFrequency settings on the client are). I've also noticed there are different versions websockets which might or might not influence things. I do hope some smartfox techy can look at this issue and see if the settings for the smartfox sockets are set correctly.

PS
The node.js server i used is from http://cjihrig.com/blog/creating-your-own-node-js-websocket-echo-server/. There is a slightly newer version of the article as mentioned above the page, i don't remember which of the two articles i used and both have the same code anyway.
User avatar
Lapo
Site Admin
Posts: 21680
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High latency when playing a game

Postby Lapo » 17 Nov 2013, 17:25

While we appreciate the amount of effort you are putting into your reporting it seems to me that the discussion is now going in too many directions.
All we need is a proof of concept of the issue, speculation will come next :)

Please refer to my previous post.
Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Lapo
Site Admin
Posts: 21680
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: High latency when playing a game

Postby Lapo » 18 Nov 2013, 16:07

Just wanted to add that a simple test sending 50 messages/sec. and receiving the same amount of from the server side, shows no changes in latency.
In any case, if you still can't figure what the problem is send us the repro-case and we'll take a look.

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
jdv145
Posts: 21
Joined: 11 May 2013, 13:13

Re: High latency when playing a game

Postby jdv145 » 19 Nov 2013, 05:59

Lapo wrote:Just wanted to add that a simple test sending 50 messages/sec. and receiving the same amount of from the server side, shows no changes in latency.
In any case, if you still can't figure what the problem is send us the repro-case and we'll take a look.

Thanks


Thanks for testing. Your results do match my expectations.... i have very little influence on the latency, it either jumps or not. It doesn't get linearly worse by increasing the load. For now i use the registry tweak on my workstation and i get very very fast responses (much better then i had hoped). Before, there was always a noticeable (but for a game like tetris acceptable) little bit of lag between controlling something and seeing your *own* action on the screen of the *opponent*. With the tweak there is no lag what so ever. Also the measured latency times are 0-2ms and no jumps. So the human perceived game experience is indeed reflected by the measured times. In short it is nothing less then perfect, but *not* on a default configured win7 station.

I tried to make a repro-case using one of the html5 examples (see attachment). I've used the tris example and added a couple of lines to the main.js to enable auto-chat (search for 'JDV' in the code to see my changes). With normal chat the latency is low. But when you give the command 'auto', the client will auto generate chat messages (5/sec) and the latency shoots up (this is escpecially the case when there are two or more clients connected). Obviously the latency depends on the client (in my case: with reg-fix: 1-10ms, without regfix: 60-110ms.. this is using the inbuild ping-pong which are printed using console.log in the js panel).

Everything (smartfox/node.js/dart/ server, html5-tris-with-autochat) is public accessible by just opening the correct url and looking at the output, so i can send the info by mail if you guys want to see it for yourselves.

PS the attachment was to big so i uploaded to the first upload site i found on google

http://www.filedropper.com/tris

Return to “SFS2X HTML5 / JavaScript API”

Who is online

Users browsing this forum: No registered users and 1 guest