Server reponse received in the wrong order

Post here your questions about the Flash / Flex / Air API for SFS2X

Moderators: Lapo, Bax

User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Server reponse received in the wrong order

Postby Evil-Dog » 27 Apr 2015, 15:11

Hi there, I'm having some issues with server communication happening in the wrong order.
Basically, I send an event when I land on a special platform in my game and another event when I jump off that platform. When I jump as soon as I land, a bug happens where I get the events back from the server in the wrong order. Here's what happens on the client.

// Landed on platform at frame 21209
SendSetOnKingPlatform true 21209
// Jump off platform one frame later at frame 21210
SendSetOnKingPlatform false 21210

The server receives this as I print it in the logs with the system time (millisecond) it was received
{KOTMTest}: OnSetOnKingPlatform false 1430146784443
{KOTMTest}: OnSetOnKingPlatform true 1430146784443
So seems like both events arrive at the same time but in the wrong order.

And so the server sends back the events to the players in the wrong order as well and the clients receives the 2 events at the same frame 21211
HandleSetOnKingPlatform false 21211
HandleSetOnKingPlatform true 21211

Is that normal? Is that a common problem, what's the best way to deal with it? I think that's the cause of many other glitch I'm having
Thank you
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 28 Apr 2015, 07:46

Yes, it is normal.
That is the essence of asynchronous programming.

It sounds like the two events, landing on the platform and jumping off, are happening very close in time on the client side. What happens is that the TCP stack on the client side aggregates the two requests into a single packet and that's why the server receives them at the same exact time.

If you need to make absolutely sure that the two requests are processed in order you will need to synchronize the threads on the server side, using a common lock.

Is this something you're familiar with?

cheers
Lapo
--
gotoAndPlay()
...addicted to flash games
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 28 Apr 2015, 12:44

Sadly I'm not familiar with that :) Can you elaborate or point to resources about the technique?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 28 Apr 2015, 13:09

Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 28 Apr 2015, 14:05

Ok I remember I've learned that at the University and it's now in the far and obscure recesses of my brain but how do I apply that to fix the problem I'm having? I *think* I understand the concepts but not how that applies to SFS handling my requests in the wrong order. Can you elaborate with where you're leading me with this? Is this the same issue and solution?
viewtopic.php?f=18&t=17721&p=76957&hilit=lock#p76957
Thanks a bunch
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 28 Apr 2015, 14:19

If you want to take a different, and possibly simpler, perspective on the problem you could try to set the Extension ThreadPool to 1.

This means that all requests are processed by a single thread avoiding parallel processing enirely. Since all messages are processed exactly in the order in which they arrive you should never see the problem happening.

This of course has side effects:
1- it doesn't allow Extensions to scale taking advantage of multiple cores
2- it can cause performance issues if your Extension calls take a long time to execute (e.g. database calls)

You could solve all of the above by creating your own Extension ThreadPool and delegating to it only the slow invocations such as database/file related calls, if any. Maybe also very computational-intense calls, such as pathfinding?

This would entirely avoid the problem of getting into the trouble of manually synchronizing threads for your use case. Come to think of it, it seems a better idea.
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 28 Apr 2015, 15:04

This is a bit out of my comfort zone but looking around to figure this out, I see a lot of people saying that their threadpool is overloaded, and causing lag, so setting the extension's threadpool size to 1 seems like a bad idea doesn't it?

Seems like any message in the wrong order can break your game state, how do huge game handle that? If sending the current weapon or setting an object state can be in the wrong order, that complicate things a lot no?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 28 Apr 2015, 15:36

Let me just clarify one point before I answer your questions.
The client requests always arrive at the server in right order if your use TCP (the default protocol). With UDP you have no guarantee for order and delivery, but I suppose we're talking about TCP here.

What happens is that they get handled by multiple threads. These threads start their work almost simultaneously but they will likely finish in different orders, because they run different code. Also if they access the same data, they will run into race conditions or make the server state logically wrong, because their parallel activity cannot be predicted deterministically.

Evil-Dog wrote:This is a bit out of my comfort zone but looking around to figure this out, I see a lot of people saying that their threadpool is overloaded, and causing lag, so setting the extension's threadpool size to 1 seems like a bad idea doesn't it?

Overloading is typically due to slow tasks, as I have mentioned. 99% of the cases it's the DB's fault.

What I suggest is to discriminate in your Extension code between code expected to be slow (e.g. database calls) vs anything else and delegate the former to a dedicated ThreadPool (which is no big deal, one line of code, just to be clear)

Seems like any message in the wrong order can break your game state, how do huge game handle that?

There can be many ways:
a) their game code doesn't break if messages come in wrong order, by design
b) they deal with the wrong order (it can be very game specific)
c) they design the game to avoid the problem entirely... (variation on a)

The bottom line is that you will need to experiment either with thread synchronization or with a mono-thread + thread pool, and see which works best for you.
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 28 Apr 2015, 17:29

Thanks for the clarification, it makes sense, threads finishing in different orders.

What I suggest is to discriminate in your Extension code between code expected to be slow (e.g. database calls) vs anything else and delegate the former to a dedicated ThreadPool (which is no big deal, one line of code, just to be clear)


That solution seems like the most logical. So I know it's not your job to teach me java or how to work with threads but are we talking about like using Executors.newSingleThreadExecutor() for the extension and Executors.newFixedThreadPool(SomeAmountOfThreads) to handle longer tasks? Or am I in the wrong direction?
I read in another SFS forum thread about creating a new runnable task to execute the longer tasks (DB access, HTTP request) in the manually created threadpool, but how does the single thread apply to the extension? That's something I haven't came across in the docs or in the forums. Maybe I'm missing some key notions about threads.
Thanks for the help
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 28 Apr 2015, 17:57

Evil-Dog wrote:Thanks for the clarification, it makes sense, threads finishing in different orders.

That solution seems like the most logical. So I know it's not your job to teach me java or how to work with threads but are we talking about like using Executors.newSingleThreadExecutor() for the extension

No the Extension thread is out of your control. You just need to configure the Extension Thread count to 1, from the AdminTool. This way all requests are executed serially.

and Executors.newFixedThreadPool(SomeAmountOfThreads) to handle longer tasks? Or am I in the wrong direction?

I would probably use and unbounded thread pool so that you can set it and forget it --> Executors.newCachedThreadPool()

Here's how it should work in simple steps:

1- you receive client requests in your code
2- you can determine whether the request should be executed by your main thread (the one you're already in) or the separate thread pool (for slow/other processes)
3- if the request should be delegated you simply hand the operation to the pool (e.g. t_pool.execute(...));
4- otherwise just execute your code.

Makes sense?
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 28 Apr 2015, 18:28

So I was trying to find an extension setting but are you talking about the global server configurator where it says Extension Thread Pool? If so, would I set Core threads to 1 and Backup threads and Maximum backups to 0? or I don't touch the backup values?
Can you only do it globally for all extensions/zones/games on the server?

Regarding newCachedThreadPool(), I thought it would be newFixedThreadPool since the doc says newCachedThreadPool() is for frequent quick tasks, but I guess it's not like DB requests will stack like crazy. Again, maybe I'm missing some important notions.

So in your workflow t_pool is the ExecutorService returned by newCachedThreadPool and I'd call like
t_pool.submit(new Runnable() {
public void run()
{
// My heavier code here.
}});
Is that right?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 29 Apr 2015, 07:12

Evil-Dog wrote:So I was trying to find an extension setting but are you talking about the global server configurator where it says Extension Thread Pool? If so, would I set Core threads to 1 and Backup threads and Maximum backups to 0? or I don't touch the backup values?
Can you only do it globally for all extensions/zones/games on the server?

Yes it is a global setting:
coreThreads = 1
backupThreads = 0
maxBackups = 0

Regarding newCachedThreadPool(), I thought it would be newFixedThreadPool since the doc says newCachedThreadPool() is for frequent quick tasks, but I guess it's not like DB requests will stack like crazy. Again, maybe I'm missing some important notions.

The cached thread will remove all headaches of re-sizing and fine tuning. I suggest this option because it's "set and forget". It will get the job done. You may replace it with something else when you're more familiar with the whole concurrency business in Java, if necessary.

So in your workflow t_pool is the ExecutorService returned by newCachedThreadPool and I'd call like
t_pool.submit(new Runnable() {
public void run()
{
// My heavier code here.
}});
Is that right?

Correct.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 29 Apr 2015, 12:40

Excellent, I'll try that and see how it goes ;)
Thanks for the help!
User avatar
Evil-Dog
Posts: 86
Joined: 08 Mar 2011, 17:59
Contact:

Re: Server reponse received in the wrong order

Postby Evil-Dog » 30 Apr 2015, 15:23

I'm unable to set the backup thread values to 0, it just resets to 1. Is that normal? It seems to fix my issue as far as my testing goes but yeah, can't set these values to 0 like you told me to.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Server reponse received in the wrong order

Postby Lapo » 30 Apr 2015, 15:48

You mean in the AdminTool?
If so, you can edit the value directly in config/server.xml (bottom of the file)

I will add a note to our bug system to check the setting in the Admin.

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X ActionScript 3 API”

Who is online

Users browsing this forum: No registered users and 11 guests