Ensure user request handing order

Post here your suggestions for new possible features in SmartFoxServer 2X.

Moderators: Lapo, Bax

abush
Posts: 13
Joined: 11 Feb 2016, 18:40

Ensure user request handing order

Postby abush » 22 Apr 2016, 21:59

I believe this is a current SFS limitation, but let me know if that assumption is incorrect.

Here's the situation I'm trying to resolve:

Code: Select all

public class Handler1 extends BaseClientRequestHandler {
    @Override
    public void handleClientRequest(User user, ISFSObject params) {
        synchronized (user) {
            user.setProperty("Handler1Called", true);
        }
    }
}

public class Handler2 extends BaseClientRequestHandler {
    @Override
    public void handleClientRequest(User user, ISFSObject params) {
        synchronized (user) {
            if(user.getProperty("Handler1Called") != null) {
                // do stuff
            }
        }
    }
}


My client sends the requests in the correct order (Handler1 then Handler2), they transfer over tcp so the server receives them in the correct order, however the requests go into separate thread pools and the handlers aren't guaranteed to be called in the correct order.

Currently the options I can come up with to resolve this are:
  • Reduce my extension thread pool size to 1
  • Create a response callback for Handler1 and have the client wait on a round trip before sending Handler2

Neither of which are ideal.

What I'd like to see is an option to have all requests from the same user routed to the same thread queue (could remove the sync blocks as well) or some other method to ensures the handlers are called in the same order the commands are received.

Thanks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Ensure user request handing order

Postby Lapo » 23 Apr 2016, 10:54

Hi,
this seems a very unique case. If the two requests need to be parsed in specific order and are sent in such a close interval that they could go out of order ... why don't you put the two requests into a single call?

In other words instead of sending reqA and then reqB as two separate requests you can send reqC which contains the parameters of both A and B and it will be executed by a handler which will run the operations in order.

Setting the Extension Executor to a single thread is too limiting in terms of scalability, so I don't recommend this.

thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
abush
Posts: 13
Joined: 11 Feb 2016, 18:40

Re: Ensure user request handing order

Postby abush » 25 Apr 2016, 18:55

Good point, for this contrived example a combination request makes sense. And in my real use case the client game play changed to injected a user step between the commands, creating the needed delay.

However, I feel there's still the potential for unexpected consequences with the current behaviour given variable server performance patterns. As another example:

  • client sends command to set a variable to value1
  • command is queued in thread A
  • thread A is starved
  • some time passed
  • client sends command to set a variable to value2
  • command is queued in thread B
  • thread B processes the 2nd command
  • thread A is unblocked
  • thread A processes the 1st command

So after this sequence the variable is set to value1 instead of value2 as the client would have expected.

Assume the server stays health and the queue processing times remain less then the client update frequency the above should be avoidable, however from a quick search I've found a couple other users requesting behaviour similar to what I'm looking for:

viewtopic.php?f=18&t=18282
viewtopic.php?f=18&t=18282#p80241
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Ensure user request handing order

Postby Lapo » 26 Apr 2016, 08:36

abush wrote:Good point, for this contrived example a combination request makes sense. And in my real use case the client game play changed to injected a user step between the commands, creating the needed delay.

However, I feel there's still the potential for unexpected consequences with the current behaviour given variable server performance patterns. As another example:

  • client sends command to set a variable to value1
  • command is queued in thread A
  • thread A is starved
  • some time passed
  • client sends command to set a variable to value2
  • command is queued in thread B
  • thread B processes the 2nd command
  • thread A is unblocked
  • thread A processes the 1st command

So after this sequence the variable is set to value1 instead of value2 as the client would have expected.

It looks like you're making some incorrect assumptions here.
In particular ---> "command is queued in thread A". Commands are not queued in a thread, they are queued in a central BlockingQueue which is monitored by a pool of threads.

In your example the first request (req A) gets picked up by the first worker thread available. If the whole thread-pool is starved (i.e. all threads are busy) ReqA will remain there until a thread is available and ReqB will be parked in the queue, behind Req A. When the next thread is available ReqA will be processed first. Then ReqB.

cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
abush
Posts: 13
Joined: 11 Feb 2016, 18:40

Re: Ensure user request handing order

Postby abush » 26 Apr 2016, 17:54

Okay, so requests are queued in the order they are received?

In that case the reason my original example is executing in a different order is because:

  • Thread A grabs a command
  • Asks for the lock on the user object
  • Thread B grabs a command
  • Asks for the lock on the user object
  • Thread B wins the lock

Which I wouldn't expect to happen very often, and only if the requests are very close to each other in the queue. However, not very often is not never, so I'll keep this in mind as a possible scenario.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Ensure user request handing order

Postby Lapo » 27 Apr 2016, 08:04

What you're describing is a very edge case, which is not even really related to how SFS works, but more generally to how Java works.

In other words: two threads contending a lock (at literally the same time) are indeterministic, in the sense that we cannot predict which thread will obtain it. We just know only one thread will, then the other.

For us to turn this into a deterministic scenario we would need to rebuild the whole server architecture from scratch from a non-blocking design to a thread-per-client solution. This involves throwing away a lot of scalability, among many other issues, to support a very specific and rare edge case...

Also this scenario would cause problems that are even worse, imho. With one dedicated thread per client (it is 2 actually, 1 for read and 1 for write) if the reading thread gets stuck in a long running call, the client will be completely frozen. Imagine a real time game, a la Unreal Tournament, players are running and dodging and shooting sending tens of positional data per second and receiving updates from all other players.

In the middle of the action the reading thread needs to grab some data from the database but the DB is very busy and the call takes 30 seconds to return... all the the other client updates fall in line, while the only active thread is waiting for the DB to wake up. The player is completely stuck for everyone else.

My suggestion is that you could implement the strict message ordering in your Extension. Number your requests and use a PriorityQueue to order them, then use your own thread pool to run the commands in the queue.

This way you can apply strict control over which command gets executed and avoid that they go out of order.

cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
abush
Posts: 13
Joined: 11 Feb 2016, 18:40

Re: Ensure user request handing order

Postby abush » 27 Apr 2016, 17:38

Fair enough.

In your scenario though if 12 users (or whatever the configured max extension thread pool size is) are all waiting on a DB operation you have the same issue but the entire game freezes, so I would discourage that kind of design in the first place.

Either way a request numbering system would allow me to detect the issue I'm concerned about.

Would it make sense to modify my feature request: to have SFS keep a counter per user (or session) that gets associated and incremented with each request, and for the request count to be available in the handleClientRequest callback?

I can work around this for now, but it seems like something that's universally useful to me.

Thanks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Ensure user request handing order

Postby Lapo » 28 Apr 2016, 06:38

abush wrote:Fair enough.

In your scenario though if 12 users (or whatever the configured max extension thread pool size is) are all waiting on a DB operation you have the same issue but the entire game freezes, so I would discourage that kind of design in the first place.

I don't think so because we use auto-scaling thread pools. The system can detect the situation you have described and add more threads on demand: http://docs2x.smartfoxserver.com/Advanc ... extensions

Would it make sense to modify my feature request: to have SFS keep a counter per user (or session) that gets associated and incremented with each request, and for the request count to be available in the handleClientRequest callback?

I can work around this for now, but it seems like something that's universally useful to me.

Yes it makes sense. I've added it to our feature request DB.

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “2X Features Wish List”

Who is online

Users browsing this forum: No registered users and 6 guests