"Extensions Controller request queue size" does not work?

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

"Extensions Controller request queue size" does not work?

Postby seeingrain » 19 Apr 2016, 05:16

Dear Admin,
We encountered problems when stress test.
We set the "Extensions Controller request queue size" to 10000, but when we doing stress test, [Dashboard->Message queues status->Extension Queue] shows there are more than 1 million Messages in queue!
So we want to know how this parameter work? when queue reaches the limit of this parameter, what will SFS do? just abandon the request? or return "server busy" like response immediately? why our message queue just keep increasing?

BTW, is it possible to expose a server API to get the current queue size to extension level? so we will know server is too busy, and just throw "server too busy, please retry later" to client?

Thank you~
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: "Extensions Controller request queue size" does not work

Postby Lapo » 19 Apr 2016, 08:06

Hi,
since the introduction of the auto-load balancing thread pools (SFS2X 2.9) we have changed how the task queue work.
Indeed the limit from the configuration is no longer enforced and those limits from the configuration could be removed.

In any case the fact that your test caused a million messages to be parked in the queue is the symptom that you have created a massive bottleneck in the system. It appears that your Extension code is running very slow calls, which I suspect are doing some very slow I/O operations.

You can configure the server to add more backup-threads as explained here:
http://docs2x.smartfoxserver.com/Advanc ... hreadModel

Keep in mind that an i/O bottleneck is not solved just by throwing more threads at it but also by optimizing the I/O operations themselves.

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

Re: "Extensions Controller request queue size" does not work

Postby seeingrain » 19 Apr 2016, 08:35

Lapo wrote:Hi,
since the introduction of the auto-load balancing thread pools (SFS2X 2.9) we have changed how the task queue work.
Indeed the limit from the configuration is no longer enforced and those limits from the configuration could be removed.

In any case the fact that your test caused a million messages to be parked in the queue is the symptom that you have created a massive bottleneck in the system. It appears that your Extension code is running very slow calls, which I suspect are doing some very slow I/O operations.

You can configure the server to add more backup-threads as explained here:
http://docs2x.smartfoxserver.com/Advanc ... hreadModel

Keep in mind that an i/O bottleneck is not solved just by throwing more threads at it but also by optimizing the I/O operations themselves.

Thanks




Yes, I know it! we are simulating a big quantity of user login and doing massive operations.
We now simulating using 6000 users to login at same time, do one operation every user every 1 second. That is to say 3000 tps. We know this is obvious heavy work for mysql even with SSD.
The actual target for us is: we want to do something CUT-DOWN machenism, when large user doing heavy works, the SFS should be robust, stable and smart, it can return "SERVER BUSY" instead of accesing DB, so that the client would retry after some delay. it can also limit the queue size to always below some size (eg: 5000 requests), so that SFS will not have to process a horrible queue like (1 million requests one by one)

That's why we were asking the question: is it possible to expose a server API to get the current queue size to extension level? so we will know server is too busy, and just throw "server too busy, please retry later" to client?
seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

Re: "Extensions Controller request queue size" does not work

Postby seeingrain » 19 Apr 2016, 08:41

Lapo wrote:Hi,
since the introduction of the auto-load balancing thread pools (SFS2X 2.9) we have changed how the task queue work.
Indeed the limit from the configuration is no longer enforced and those limits from the configuration could be removed.

In any case the fact that your test caused a million messages to be parked in the queue is the symptom that you have created a massive bottleneck in the system. It appears that your Extension code is running very slow calls, which I suspect are doing some very slow I/O operations.

You can configure the server to add more backup-threads as explained here:
http://docs2x.smartfoxserver.com/Advanc ... hreadModel

Keep in mind that an i/O bottleneck is not solved just by throwing more threads at it but also by optimizing the I/O operations themselves.

Thanks


No matter how many load-balance threads running, the bottle neck always exist, because DB's capacity is limited, no matter how we optimize the slow query.
All we want is to implement something like "FUSE" or "AIR SWITCH", when ampere is too big, just skip/ignore future requests, and recover to work when task queue becomes small automatically.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: "Extensions Controller request queue size" does not work

Postby Lapo » 19 Apr 2016, 09:35

seeingrain wrote:No matter how many load-balance threads running, the bottle neck always exist, because DB's capacity is limited, no matter how we optimize the slow query.

But why is that?
Databases are built to scale and hardware can be improved if that's the main cause of your bottleneck.

The fact that you're reaching millions of queued operations is a big red flag that the system is not going to function properly. Why would you not try to improve it?

All we want is to implement something like "FUSE" or "AIR SWITCH", when ampere is too big, just skip/ignore future requests, and recover to work when task queue becomes small automatically.

But the problem here is that you're breaking many user's experience. If requests are dropped the client's behavior will fail.
Your "fuse" analogy means that if you have 100 people working at their computers all of sudden 50 of them will have their laptops shut off and loose all their work.

I am not sure I understand where you're going with this idea, and maybe it's none of my business but I'd like to suggest the optimal way to deal with such a bottleneck, if you allow me. :) Otherwise I will shut up. :)

From a SmartFoxServer API standpoint we don't expose the lower level API because you can break things down if you don't know what you're doing.
If you want to get to the thread pool executor you'll need something like this:

Code: Select all

SmartThreadPoolExecutor executor = (SmartThreadPoolExecutor) SmartFoxServer.getInstance().getEventManager().getThreadPool();

Then you can access the queue and it's size.

The problem is that even when you know the queue size there is not much you can do with it, besides maybe refusing more user logins.
If this is the solution then you don't need to mess with lower level queues to block users from logging in. You can very easily determine how many users will generate enough traffic to hit the bottleneck and then simply limit your Zone to that number of Users.
The server will do the rest, by simply refusing more than X amount of users, where X can be 100, 1000 or whatever number of users will cause the bottleneck.

Hope I made mysefl clear.
Lapo

--

gotoAndPlay()

...addicted to flash games
seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

Re: "Extensions Controller request queue size" does not work

Postby seeingrain » 19 Apr 2016, 09:51

Lapo wrote:
seeingrain wrote:No matter how many load-balance threads running, the bottle neck always exist, because DB's capacity is limited, no matter how we optimize the slow query.

But why is that?
Databases are built to scale and hardware can be improved if that's the main cause of your bottleneck.

The fact that you're reaching millions of queued operations is a big red flag that the system is not going to function properly. Why would you not try to improve it?

All we want is to implement something like "FUSE" or "AIR SWITCH", when ampere is too big, just skip/ignore future requests, and recover to work when task queue becomes small automatically.

But the problem here is that you're breaking many user's experience. If requests are dropped the client's behavior will fail.
Your "fuse" analogy means that if you have 100 people working at their computers all of sudden 50 of them will have their laptops shut off and loose all their work.

I am not sure I understand where you're going with this idea, and maybe it's none of my business but I'd like to suggest the optimal way to deal with such a bottleneck, if you allow me. :) Otherwise I will shut up. :)

From a SmartFoxServer API standpoint we don't expose the lower level API because you can break things down if you don't know what you're doing.
If you want to get to the thread pool executor you'll need something like this:

Code: Select all

SmartThreadPoolExecutor executor = (SmartThreadPoolExecutor) SmartFoxServer.getInstance().getEventManager().getThreadPool();

Then you can access the queue and it's size.

The problem is that even when you know the queue size there is not much you can do with it, besides maybe refusing more user logins.
If this is the solution then you don't need to mess with lower level queues to block users from logging in. You can very easily determine how many users will generate enough traffic to hit the bottleneck and then simply limit your Zone to that number of Users.
The server will do the rest, by simply refusing more than X amount of users, where X can be 100, 1000 or whatever number of users will cause the bottleneck.

Hope I made mysefl clear.


Ya, we gonna make a try to check the thread pool size.
The reason we do this is: in realistic scenario, we cannot predict user's action in production environment, eg:
1. when openning new server, many users are waiting for the opening at 9:00AM, so at 9 AM, it will come a huge login request. if we can deny some login request at short duration, our server wouldn't go crazy, use up memory... the client will automatically login again after some period. however, in the client UI, the client just shows "login, please wait...", user does not know server busy event at all.
2. when we make a discount in shop, many user will perform a rush buy, we cannot predict how many people rushing, so we cannot limit the zone's user count.

in my opinion, returning server busy is rough, but it is easy to implement and keeps our server safe (not crashing) during production run-time. of course we can do something queuing user request due to different request, but time is not so rich~~~
seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

Re: "Extensions Controller request queue size" does not work

Postby seeingrain » 19 Apr 2016, 10:13

seeingrain wrote:
Lapo wrote:
seeingrain wrote:No matter how many load-balance threads running, the bottle neck always exist, because DB's capacity is limited, no matter how we optimize the slow query.

But why is that?
Databases are built to scale and hardware can be improved if that's the main cause of your bottleneck.

The fact that you're reaching millions of queued operations is a big red flag that the system is not going to function properly. Why would you not try to improve it?

All we want is to implement something like "FUSE" or "AIR SWITCH", when ampere is too big, just skip/ignore future requests, and recover to work when task queue becomes small automatically.

But the problem here is that you're breaking many user's experience. If requests are dropped the client's behavior will fail.
Your "fuse" analogy means that if you have 100 people working at their computers all of sudden 50 of them will have their laptops shut off and loose all their work.

I am not sure I understand where you're going with this idea, and maybe it's none of my business but I'd like to suggest the optimal way to deal with such a bottleneck, if you allow me. :) Otherwise I will shut up. :)

From a SmartFoxServer API standpoint we don't expose the lower level API because you can break things down if you don't know what you're doing.
If you want to get to the thread pool executor you'll need something like this:

Code: Select all

SmartThreadPoolExecutor executor = (SmartThreadPoolExecutor) SmartFoxServer.getInstance().getEventManager().getThreadPool();

Then you can access the queue and it's size.

The problem is that even when you know the queue size there is not much you can do with it, besides maybe refusing more user logins.
If this is the solution then you don't need to mess with lower level queues to block users from logging in. You can very easily determine how many users will generate enough traffic to hit the bottleneck and then simply limit your Zone to that number of Users.
The server will do the rest, by simply refusing more than X amount of users, where X can be 100, 1000 or whatever number of users will cause the bottleneck.

Hope I made mysefl clear.


Ya, we gonna make a try to check the thread pool size.
The reason we do this is: in realistic scenario, we cannot predict user's action in production environment, eg:
1. when openning new server, many users are waiting for the opening at 9:00AM, so at 9 AM, it will come a huge login request. if we can deny some login request at short duration, our server wouldn't go crazy, use up memory... the client will automatically login again after some period. however, in the client UI, the client just shows "login, please wait...", user does not know server busy event at all.
2. when we make a discount in shop, many user will perform a rush buy, we cannot predict how many people rushing, so we cannot limit the zone's user count.

in my opinion, returning server busy is rough, but it is easy to implement and keeps our server safe (not crashing) during production run-time. of course we can do something queuing user request due to different request, but time is not so rich~~~



Would you mind detail describe how to get the current message queue size? we tried but not so sure which function returns the expected result...
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: "Extensions Controller request queue size" does not work

Postby Lapo » 19 Apr 2016, 10:33

Code: Select all

SmartThreadPoolExecutor executor = (SmartThreadPoolExecutor) SmartFoxServer.getInstance().getEventManager().getThreadPool();
int size = executor.getQueue().size()


Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

Re: "Extensions Controller request queue size" does not work

Postby seeingrain » 19 Apr 2016, 11:20

Lapo wrote:

Code: Select all

SmartThreadPoolExecutor executor = (SmartThreadPoolExecutor) SmartFoxServer.getInstance().getEventManager().getThreadPool();
int size = executor.getQueue().size()


Cheers


Thank you, we will make a try
seeingrain
Posts: 28
Joined: 27 Mar 2014, 03:34

Re: "Extensions Controller request queue size" does not work

Postby seeingrain » 20 Apr 2016, 03:38

Lapo wrote:

Code: Select all

SmartThreadPoolExecutor executor = (SmartThreadPoolExecutor) SmartFoxServer.getInstance().getEventManager().getThreadPool();
int size = executor.getQueue().size()


Cheers


Dear Lapo... we were able to fetch the queue size now. thank you~
But now, we block user request when handle client request callback is called. Is it possible we do this when adding queue? using SFSExtensionFilter?
If your answer is yes, pls do us a favor to briefly describe the outline of how to use SFSExtensionFilter, we cannot find demo in help docs.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: "Extensions Controller request queue size" does not work

Postby Lapo » 20 Apr 2016, 07:53

seeingrain wrote:
Lapo wrote:But now, we block user request when handle client request callback is called.

I am not sure I really understand what you're trying to do...

When the handleClientRequest() is being called in your Extension the client's request has already made it into the main Users' request queue and was finally picked up by one of the worker threads which is now executing it via that method (handleClientRequest).

Can you clarify what you're trying to do?
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 37 guests