Read queue under heavy load

Need help with SmartFoxServer? You didn't find an answer in our documentation? Please, post your questions here!

Moderators: Lapo, Bax

cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

Read queue under heavy load

Postby cnPauly » 13 May 2009, 16:35

The actual trace output is

[ WARNING ] > Read queue under heavy load (121 msgs)

Is this warning referring to the Incoming queue?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 14 May 2009, 07:27

Hi,
this message is rarely seen. Does it appears many times in the logs?
What server version do you use?
Lapo
--
gotoAndPlay()
...addicted to flash games
cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

Postby cnPauly » 14 May 2009, 11:29

I'm using ver 1.6.5.01 - linux 64 bit server jvm assigned 500 mb of heap - running on a 2.6ghz dual core with plenty of ram.

(the production machine will be substantially more powerful - this machine is for benchmarking and testing the game under load)

The config I'm using

<OutQueueThreads>2</OutQueueThreads>
<ExtHandlerThreads>1</ExtHandlerThreads>

(I'm thinking of assigning 2 threads to the exthandler and 1 to out q)


<MaxWriterQueue>500</MaxWriterQueue>

<MaxIncomingQueue>20000</MaxIncomingQueue>
<DeadChannelsPolicy>strict</DeadChannelsPolicy>
<MaxMsgLen>4000</MaxMsgLen>

<ClientMessagQueue>
<QueueSize>250</QueueSize>
<MaxAllowedDroppedPackets>1</MaxAllowedDroppedPackets>
</ClientMessagQueue>



I'm getting these intermittently, sometimes one or two, sometimes a bunch at once. When it happens the dropped in messages increase.

I have built a stress testing app using the java client api. The app works just as you have suggested in other threads - I load 500 clients and have them simulate game play messaging.

Right now we have it set up to where every 2 player game room loads a room extension that handles the messaging for that game. Every client sends a 22 byte string xt message to its room extension every 60 milliseconds.

The room extension sends a 40 byte string xt message back to each client every 60 milliseconds. These are not synchronized in the sense that the room xt does not wait to get client messages before it sends. These are just sending and receiving at will.

I cranked up 3 Stress apps (1500 connections) on 3 different computers this morning all hitting the server through a lan. They seem to be running ok. There are no dropped out-messages but there is 40k of dropped in-messages - which is weired right?

I'm sending only 22 bytes and the maxmsglen is set to 4k. huh....
cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

Postby cnPauly » 14 May 2009, 11:40

in case it matters I'm running the server using the wrapper.

it also might be worth mentioning that I'm calling handleInternalRequest on each room xt every 60ms from a game manager class instanced from the zone ext, as a method to call an update tik that sends the outgoing message.

I actually have 50 game managers each responsible to update x number of games. This keeps the tread count down.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 14 May 2009, 12:37

They seem to be running ok. There are no dropped out-messages but there is 40k of dropped in-messages - which is weired right?

Dropped incoming messages are simply caused by the size of the message, in fact the server discards any message whose length is greater than the size specified in the <MaxMsgLen> setting (config.xml)

By default that size is == 4096 bytes (4Kb) but you can adjust it according to your needs.
I'm sending only 22 bytes and the maxmsglen is set to 4k. huh....

The stress client is probably misbehaving... you should take a look at which messages are going through the network. Each messages should be terminated by 0x00 byte.
You can user WireShark or a similar tool to capture the traffic.
Also you can make SFS output debug data about each incoming/outgoing message. Check Chapter 2.1 of the docs.
Lapo

--

gotoAndPlay()

...addicted to flash games
cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

Postby cnPauly » 14 May 2009, 12:53

Dropped incoming messages are simply caused by the size of the message, in fact the server discards any message whose length is greater than the size specified in the <MaxMsgLen> setting (config.xml)


right - I have it set to 4kb and the total message size for each message is only 100 or so bytes.

I swapped the number of threads to;
<OutQueueThreads>1</OutQueueThreads>
<ExtHandlerThreads>2</ExtHandlerThreads>

and now I get no dropped in messages but I still get this (taken from the admin tool log)

[ WARNING ] [id: 18] (SmartFoxServer.readIncomingMessages): Read queue under heavy load (102 msgs)

102 msgs is heavy load?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 14 May 2009, 13:00

That's huge :)
Because that's one read cycle ... which happens every few nanosconds. 100 messages queued in a few nanoseconds is very much.
Do you monitor the bandwidth occupied by the test? If you saturate it you might get strange behaviors especially if the clients are not keeping up with the server responses.
Lapo

--

gotoAndPlay()

...addicted to flash games
cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

More Observations

Postby cnPauly » 14 May 2009, 13:27

Each Stress App is handling roughly 1.2 Mbs out and 1 Mbs in.

I am getting some strange messages occasionally -

Using wireshark I traced a set of messages coming from a Stress app running just 2 clients. (this is while I have 1500 clients already running on other machines)

Normal Message from client to server
Data = 25787425636e47616d652565762537353225353735332500
(%xt%cnGame%ev%752%5753%)

Then - once in a while it doubles or triples up

Data = 25787425636e47616d65256576253735322535373535250025787425636e4
7616d65256576253735322535373536250025787425636e47616d65256576
2537353225353735372500
(%xt%cnGame%ev%752%5753%.
%xt%cnGame%ev%752%5753%.
%xt%cnGame%ev%752%5753%)

they all seem to be terminating with 0x00 byte though.

Each client is using the simplest broadcast method possible.

int packetId=0;
public void dataSend() {
packetId++;
String[] o = { Integer.toString(packetId) };
sendXtMessage("cnGame", "ev", o, _gameId);
}

By the way - I'm using the Mina xml socket version of the java client api.
cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

Could it be the client socket classes

Postby cnPauly » 15 May 2009, 11:27

It looks to me that every once in a while the socket is not properly sending the messages individually and instead packing many messages (up to four so far) into one tcp packet.

So - the client is definitely misbehaving but I'm not sure I can do anything in my code to fix it - I think it's an issue with the socket classes.

My thinking is that this might explain whats going on here.

If the tcp packet is filled up with multiple messages instead of flushing after a single message - would the incoming queue possibly reject it because of size or is it unpacked into individual messages before checking for size. (13.9k dropped in on my last test)

It appears however that smartfox is unpacking the messages correctly - whenever it happens - because I haven't been able to find a case of multiple messages when I trace the incoming using the debug incoming messages flag.

couldn't that scenario cause the incoming queue to flood?


Any thoughts?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 15 May 2009, 15:09

It appears however that smartfox is unpacking the messages correctly - whenever it happens - because I haven't been able to find a case of multiple messages when I trace the incoming using the debug incoming messages flag.

Yes that's the reason of the error message you get.
The server receives a single packet with many requests aggregated so he's amazed at how many messages he got in one shot :)
Anyways that warning is negligible and simply indicates that requests are handled correctly on the server side.

HOWEVER... if I guess that if the aggregated messages exceed the size of 4Kb then the server will refuse it...

So the simple solution is to allow the server "swallow" larger messages such as 64Kb or 128Kb. In production I wouldn't suggest to go past 32-64Kb, and also it's very unlikely that this message aggregation is performed over the internet (it might happen, but definitely not so aggressively, it depends on the so-called Nagle algorithm)
Lapo

--

gotoAndPlay()

...addicted to flash games
cnPauly
Posts: 13
Joined: 08 Apr 2009, 18:32

Postby cnPauly » 15 May 2009, 15:40

thanks for the support - helps a great deal

As I study my wireshark throughput I notice that the aggregation always happens on a TCP Retransmission - sometimes aggregating 10 or more messages in one packet.
User avatar
yuvallahav
Posts: 247
Joined: 07 Oct 2008, 13:03
Location: Rome, Italy
Contact:

Re: Read queue under heavy load

Postby yuvallahav » 26 Sep 2012, 14:05

Sorry to wake a sleeping dragon, but I'm facing the same warning message on our production server these past couple of weeks, it got a little bit better when I've set the max incoming message length to 32Kb, which helped it a little but, but I'm still getting this message.
Anything else I can do on the client side to help it relive the pressure??
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Read queue under heavy load

Postby Lapo » 26 Sep 2012, 14:58

I would suggest to check the status of the server machine, and the CPU consumed by the Java process.
That's usually the cause.
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SmartFoxServer 1.x Discussions and Help”

Who is online

Users browsing this forum: No registered users and 30 guests