Information about Thread Pool size

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Information about Thread Pool size

Postby pratts » 22 Jan 2018, 05:20

Hello,
We have the default configuration of system thread pool and extension thread pool. I would like to know, the default number of threads created at runtime during server start. Does it depend upon the number of CPU cores ?

Code: Select all

<systemThreadPoolSettings>
    <name>Sys</name>
    <coreThreads>4</coreThreads>
    <backupThreads>8</backupThreads>
    <maxBackups>16</maxBackups>
    <queueSizeTriggeringBackup>100</queueSizeTriggeringBackup>
    <secondsTriggeringBackup>50</secondsTriggeringBackup>
    <backupThreadsExpiry>3600</backupThreadsExpiry>
    <queueSizeTriggeringBackupExpiry>60</queueSizeTriggeringBackupExpiry>
    <logActivity>true</logActivity>
    <queueFullWarningInterval>300</queueFullWarningInterval>
  </systemThreadPoolSettings>
  <extensionThreadPoolSettings>
    <name>Ext</name>
    <coreThreads>4</coreThreads>
    <backupThreads>8</backupThreads>
    <maxBackups>16</maxBackups>
    <queueSizeTriggeringBackup>100</queueSizeTriggeringBackup>
    <secondsTriggeringBackup>50</secondsTriggeringBackup>
    <backupThreadsExpiry>3600</backupThreadsExpiry>
    <queueSizeTriggeringBackupExpiry>60</queueSizeTriggeringBackupExpiry>
    <logActivity>true</logActivity>
    <queueFullWarningInterval>300</queueFullWarningInterval>
  </extensionThreadPoolSettings>
 


We changes the extension thread pool from 4 to 8, but it only changed the number from 4 to 5 in smartfox message queue thread count.

I also want to know the purpose of the the line mentioned line in core.xml file:

Code: Select all

<sessionPacketQueueSize>160</sessionPacketQueueSize>


Thank you.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 22 Jan 2018, 10:54

Hi,
I would not recommend changing any of those settings unless you know what you're doing and, more importantly, you really need it.
Is there a specific reason why you need to change those?

SFS2X runs auto-scaling thread pools to avoid manual tweaking. Read here:
http://docs2x.smartfoxserver.com/Extens ... hreadModel

Hope it helps
Lapo
--
gotoAndPlay()
...addicted to flash games
pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Re: Information about Thread Pool size

Postby pratts » 23 Jan 2018, 08:00

Hello,
We are facing a problem with client getting disconnected suddenly.
On reconnection, there is a problem when client tries to login again.
We are trying to figure out the problem on client's network, if any.
But we are just trying to find out the meaning of the parameters :

Code: Select all

<!-- Max size of incoming packet -->
   <maxIncomingRequestSize>1000000</maxIncomingRequestSize>
   
   <!-- Max Size for read buffer -->
   <maxReadBufferSize>1024</maxReadBufferSize>
   
   <!-- Max Size for write buffer -->
   <maxWriteBufferSize>32768</maxWriteBufferSize>
   
   <!-- Session Queue Size -->
   <sessionPacketQueueSize>160</sessionPacketQueueSize>
   
   <!-- Thread Pool Size of the 3 main stages of the Bitswarm Engine -->
   <socketAcceptorThreadPoolSize>1</socketAcceptorThreadPoolSize>
   <socketReaderThreadPoolSize>1</socketReaderThreadPoolSize>
   <socketWriterThreadPoolSize>3</socketWriterThreadPoolSize>


We are just trying to check whether the pool size we have is enough (on both extension and system end) ?
We also want to know the number of threads smartfox uses for connection handling and the inbound and outbound message queue lengths.

We also found some of the threads for which we want some of the details :

Code: Select all

"RMI TCP Connection(2)-127.0.0.1" daemon prio=10 tid=0x00007f76f000d800 nid=0x48e2 runnable [0x00007f77098d3000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.io.BufferedInputStream.fill(Unknown Source)
   at java.io.BufferedInputStream.read(Unknown Source)
   - locked <0x00000007645bded0> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(Unknown Source)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - <0x00000007643e5df0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"JMX server connection timeout 88" daemon prio=10 tid=0x00007f76ec0d1800 nid=0x48df in Object.wait() [0x00007f77099d4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x000000076447ebb8> (a [I)
   at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
   - locked <0x000000076447ebb8> (a [I)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"RMI Scheduler(0)" daemon prio=10 tid=0x00007f76ec0cb000 nid=0x48de waiting on condition [0x00007f7709ad5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007643e0378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"SFSWorker:Ext:4" prio=10 tid=0x00007f76c8024000 nid=0x44f8 waiting on condition [0x00007f771c699000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000774ce18f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(Unknown Source)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
   at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"pool-11-thread-2" prio=10 tid=0x00007f76b0018000 nid=0x442a waiting on condition [0x00007f771c99c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000772dd72c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon prio=10 tid=0x00007f76e01c9800 nid=0x4427 in Object.wait() [0x00007f771cc9f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x000000077599cff8> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
   - locked <0x000000077599cff8> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

   Locked ownable synchronizers:
   - None

"Timer-1" daemon prio=10 tid=0x00007f76e015c000 nid=0x4424 in Object.wait() [0x00007f771cfa2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000000775a37010> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Unknown Source)
   - locked <0x0000000775a37010> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"SFSWorker:Sys:3" prio=10 tid=0x00007f76d0002800 nid=0x4423 waiting on condition [0x00007f771d0a3000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000774ce17a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(Unknown Source)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
   at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"Scanner-0" daemon prio=10 tid=0x00007f772d24e000 nid=0x43e7 in Object.wait() [0x00007f771d3a6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00000007759ce258> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Unknown Source)
   - locked <0x00000007759ce258> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"qtp2093568729-55" prio=10 tid=0x00007f772d146000 nid=0x43e3 waiting on condition [0x00007f771d7aa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000775aed8c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
   at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"qtp2093568729-50-selector-ServerConnectorManager@24b027dd/1" prio=10 tid=0x00007f772d13c000 nid=0x43de runnable [0x00007f771dcaf000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
   at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
   - locked <0x0000000775aeb0a0> (a sun.nio.ch.Util$2)
   - locked <0x0000000775aeb028> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000000774d65ba0> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(Unknown Source)
   at sun.nio.ch.SelectorImpl.select(Unknown Source)
   at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
   at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None

"New I/O server boss #9" prio=10 tid=0x00007f772cf77800 nid=0x43d2 runnable [0x00007f771e9c0000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
   at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
   - locked <0x00000007754ff898> (a sun.nio.ch.Util$2)
   - locked <0x00000007754ff8b0> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000000774eda770> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(Unknown Source)
   at sun.nio.ch.SelectorImpl.select(Unknown Source)
   at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
   at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
   at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
   at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
   at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - <0x0000000774eeebe0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon prio=10 tid=0x00007f772cce8800 nid=0x43c6 in Object.wait() [0x00007f771f5cc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000000774d36600> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
   - locked <0x0000000774d36600> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

   Locked ownable synchronizers:
   - None


Could you please provide some insights on the above mentioned points ?

Thank you
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 23 Jan 2018, 09:23

pratts wrote:Hello,
We are facing a problem with client getting disconnected suddenly.

Is it a single client problem? Or do you have reports from multiple people of the same issue?

We are just trying to check whether the pool size we have is enough (on both extension and system end) ?

The best way to do this is by monitoring the state of the server's queues from the AdminTool > Dashboard.
If your thread pools are not keeping up you will see the queues being full for long periods of time, while normally you should just see a few spikes every now and then.

We also want to know the number of threads smartfox uses for connection handling and the inbound and outbound message queue lengths.

It depends on the load. Have you read the article I linked in the previous post?
http://docs2x.smartfoxserver.com/Extens ... hreadModel
This should clarify how thread pools work.


We also found some of the threads for which we want some of the details :
"RMI TCP Connection(2)-127.0.0.1" daemon prio=10 tid=0x00007f76f000d800 nid=0x48e2 runnable [0x00007f77098d3000]

"JMX server connection timeout 88" daemon prio=10 tid=0x00007f76ec0d1800 nid=0x48df in Object.wait() [0x00007f77099d4000]


"RMI Scheduler(0)" daemon prio=10 tid=0x00007f76ec0cb000 nid=0x48de waiting on condition [0x00007f7709ad5000]

These are extraneous threads. Very likely they are from VisualVM or equivalent JMX tool.

"SFSWorker:Ext:4" prio=10 tid=0x00007f76c8024000 nid=0x44f8 waiting on condition [0x00007f771c699000]

This is an Extension worker.
"pool-11-thread-2" prio=10 tid=0x00007f76b0018000 nid=0x442a waiting on condition [0x00007f771c99c000]

Probably a scheduler's thread

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon prio=10 tid=0x00007f76e01c9800 nid=0x4427

No idea, this is not from SmartFox. Maybe your code, or your libraries.

"Timer-1" daemon prio=10 tid=0x00007f76e015c000 nid=0x4424 in Object.wait() [0x00007f771cfa2000]

A Timer thread. Not from SFS2X. We don't use timers.

"SFSWorker:Sys:3" prio=10 tid=0x00007f76d0002800 nid=0x4423 waiting on condition [0x00007f771d0a3000]

Another SFS2X worker.

"Scanner-0" daemon prio=10 tid=0x00007f772d24e000 nid=0x43e7 in Object.wait() [0x00007f771d3a6000]

Extension deploy monitor.

"qtp2093568729-55" prio=10 tid=0x00007f772d146000 nid=0x43e3 waiting on condition [0x00007f771d7aa000]

"qtp2093568729-50-selector-ServerConnectorManager@24b027dd/1" prio=10 tid=0x00007f772d13c000 nid=0x43de runnable

"New I/O server boss #9" prio=10 tid=0x00007f772cf77800 nid=0x43d2 runnable [0x00007f771e9c0000]

Jetty's threads.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Re: Information about Thread Pool size

Postby pratts » 23 Jan 2018, 09:37

Thank you for replying
Is it a single client problem? Or do you have reports from multiple people of the same issue?

This problem is happening with multiple client. Not all the client. Also the timings of disconnection vary.

The best way to do this is by monitoring the state of the server's queues from the AdminTool > Dashboard.
If your thread pools are not keeping up you will see the queues being full for long periods of time, while normally you should just see a few spikes every now and then.

We saw a spike for a moment in system workload,extension queue, outgoing message queue.

We also want to know the number of threads smartfox uses for connection handling and the inbound and outbound message queue lengths.

What we want to know if smartfox maintains another thread pool to accept user connection or system thread pool is used for that.
As far as I know, it is maintaining 3 thread pools :
1. System thread pool defined in systemThreadPoolSettings
2. Extension thread pool defined in extensionThreadPoolSettings
3. Scheduler thread pool defined in schedulerThreadPoolSize used as

Code: Select all

SmartFoxServer.getInstance().getTaskScheduler()


I saw the following lines in core.xml :

Code: Select all

<socketAcceptorThreadPoolSize>1</socketAcceptorThreadPoolSize>
   <socketReaderThreadPoolSize>1</socketReaderThreadPoolSize>
   <socketWriterThreadPoolSize>3</socketWriterThreadPoolSize>   

What are these parameters ? Do they have anything to do with a thread pool accepting the connections from client and processing accordingly ?
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 23 Jan 2018, 09:59

pratts wrote:Thank you for replying
This problem is happening with multiple client. Not all the client. Also the timings of disconnection vary.

It is expected that a minor portion of clients will have connection problems. If this represents a ~5-6% of the cases it's normal.

We saw a spike for a moment in system workload,extension queue, outgoing message queue.

Short spikes are of no concern. Flat lines or upward slopes are, because they indicate that the server is constantly behind the workload.

We also want to know the number of threads smartfox uses for connection handling and the inbound and outbound message queue lengths.

The connection acceptor is a single thread. There is no need to multiple threads, the operation is extremely fast.
Outbound queue: every user has his outbound queue. This however has little to no relationship with connection issues.
Inbound queue: there is currently no limit, so the limit is the JVM memory. Again I can't see the link with connection issues.

What we want to know if smartfox maintains another thread pool to accept user connection or system thread pool is used for that.

There is a single dedicated thread for accepting connections.
As far as I know, it is maintaining 3 thread pools :
1. System thread pool defined in systemThreadPoolSettings
2. Extension thread pool defined in extensionThreadPoolSettings
3. Scheduler thread pool defined in schedulerThreadPoolSize used as

Code: Select all

SmartFoxServer.getInstance().getTaskScheduler()


In essence this is correct. There are more pools but they are used for other tasks, such as scheduling etc...

I saw the following lines in core.xml :

Code: Select all

<socketAcceptorThreadPoolSize>1</socketAcceptorThreadPoolSize>
   <socketReaderThreadPoolSize>1</socketReaderThreadPoolSize>
   <socketWriterThreadPoolSize>3</socketWriterThreadPoolSize>   

What are these parameters ? Do they have anything to do with a thread pool accepting the connections from client and processing accordingly ?[/quote]

No and I can tell you with great confidence that that thread pools are not going to affect the ability of client to connect.
Instead problems with logins can be cause by Extension code holding threads for too long thus starving the thread pool and causing the client to timeout before the transaction is complete. But that's an entirely different story.

Does this help?
Lapo

--

gotoAndPlay()

...addicted to flash games
pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Re: Information about Thread Pool size

Postby pratts » 23 Jan 2018, 12:39

Thank you for the feedback.
It is mentioned in the documentation :
SmartFoxServer 2X runs all Extensions in a multithreaded environment. There are fundamentally two separate thread pools operating on an Extension: the ExtensionController and the SystemController. The former entity is responsible for processing client requests while the latter dispatches system events such as LOGIN, USER_DISCONNECT, ROOM_VARIABLES_UPDATE, etc.


I saw the line

Code: Select all

23 Jan 2018 | 18:00:13,724 | INFO  | SFSWorker:Ext:3 | v2.api.SFSApi |     | User login: { Zone: PokerKing }, ( User Name: abc, Id: 0, Priv: 0, Sess: 127.0.0.1:54146 ) , Type: JavaScript
in smartfox logs for login.

Is it the extension thread ? If yes, then it is occupying the extension thread pool and other user related requests and it differs from the documentation.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 23 Jan 2018, 15:03

SFSWorker:Ext is an extension thread.
SFSWorker:Sys is a system thread.

If yes, then it is occupying the extension thread pool and other user related requests and it differs from the documentation.

No, I would say that the documentation is accurate.
The Login operation is a multi-phase operation that involves both System and Ext threads, though most of the work is done by the system thread.

Still... I am not too sure why we're fixating on threads when the problem has to do with a sudden disconnection.
There is little to no relationship between threading issues and a sudden disconnection.

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Re: Information about Thread Pool size

Postby sitnik » 24 Jan 2018, 08:55

Hi Lapo,

I fully agree with your point that your Threadpool is tuned enough to handle most of the requests load.
Actually we are asking about increasing Thread pool size of extension requests with the perspective of reducing response time. As delay in response is causing issues to our reconnection logic.

So would you suggest to increase Threadpool size if we have got a Strong server machine with lots of cores. I hope it wont cause any issues and reduce our response time of requests, however at the same time we will also identify the cause of delay causing tasks in the extension code.

Waiting for your positive and quick feedback !!
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 24 Jan 2018, 09:46

Hello,
sitnik wrote:Hi Lapo,

I fully agree with your point that your Threadpool is tuned enough to handle most of the requests load.
Actually we are asking about increasing Thread pool size of extension requests with the perspective of reducing response time. As delay in response is causing issues to our reconnection logic.

How did you determine that this is the cause?
Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 24 Jan 2018, 09:52

Also can you explain how your "reconnection logic" works. I thought that when we were talking about "reconnection" we were talking about the standard reconnection system in SFS2X.

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Re: Information about Thread Pool size

Postby sitnik » 24 Jan 2018, 10:43

Lapo wrote:Hello,
sitnik wrote:Hi Lapo,

I fully agree with your point that your Threadpool is tuned enough to handle most of the requests load.
Actually we are asking about increasing Thread pool size of extension requests with the perspective of reducing response time. As delay in response is causing issues to our reconnection logic.

How did you determine that this is the cause?
Thanks



See we have built our own logic for detecting disconnection from server on the client side []say network fluctuation, outage etc.]. We do use SFS events as well however we have experienced SFS takes a little more time for detecting disconnection [Dont mind], so we have another addon feature to detect the disconnection on client.

Its like a heart beat between server & client, it works this way:

Client send heathbeat to server every 5 seconds, and server revert back with a response the moment it recieves heartbeat request.
If the client finds that last 3 responses havent arrived then we call it a disconnection scenario and run our reconnection logic [again its an addon feature we have built], we find it better than SFS reconnection support [Dont Mind] .

So now because the SFS request threads are busy doing some task, they are not able to process Heartbeat request instantly which results into some response time, sometimes it exceeds 15 seconds, meanwhile our client considers it as disconnection, which is causing unnecessary disconnection/reconnection restorations on client view.

This is how it works.
sitnik
Posts: 19
Joined: 24 Jan 2018, 08:49

Re: Information about Thread Pool size

Postby sitnik » 24 Jan 2018, 10:47

Lapo wrote:Also can you explain how your "reconnection logic" works. I thought that when we were talking about "reconnection" we were talking about the standard reconnection system in SFS2X.

Thanks


Lapo, its not the same, in our addon feature we reattempt connection + login N times with some time gap, say 15 seconds.
Also we reinitialize sfs instance on client side and all the listeners as well.

Thats how it works.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Information about Thread Pool size

Postby Lapo » 24 Jan 2018, 14:38

sitnik wrote:Its like a heart beat between server & client, it works this way:

Client send heathbeat to server every 5 seconds, and server revert back with a response the moment it recieves heartbeat request.
If the client finds that last 3 responses havent arrived then we call it a disconnection scenario and run our reconnection logic [again its an addon feature we have built], we find it better than SFS reconnection support [Dont Mind] .

So now because the SFS request threads are busy doing some task, they are not able to process Heartbeat request instantly which results into some response time, sometimes it exceeds 15 seconds, meanwhile our client considers it as disconnection, which is causing unnecessary disconnection/reconnection restorations on client view.

This is how it works.

The fact that the server is not able to reply in 15 seconds is really bad.
I would say that if it's the server fault then you should be seeing massive backlog of work in the server queues.

However, from the client perspective it is really difficult to say what caused the last 3 heartbeats to be delayed. It could be due to the connection itself, and therefore starting a reconnection would be counterproductive, since the network is already in a bad state.

The question you haven't clarified here is... how do you know the server is lagging for 15+ seconds? And what would be the cause of that?
I mean, SFS2X doesn't lag like that unless some Extensions are occupying all threads for a long time.

Do you heavily rely on long running I/O operations in your server side code? Any chance you could improve such big latency? (e.g. talking to servers in a local network rather than remotely?)

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
balamost
Posts: 9
Joined: 08 May 2017, 10:29

Re: Information about Thread Pool size

Postby balamost » 16 Apr 2019, 12:52

Hi Lapo,

I have some issues with config pool size,
Sometimes, My server game has lagged a bit
When I check the "Dashboard/Message Queue status" in the admin tool
I saw the "Extension Queue" seem to reach maximum (58/64)
So, could I increase "Extension thread pool" like this:
- Core threads: 8
- Backup threads: 32
- Maximum backup: 128
P/S: current config is the default value (4 - 16 - 64)
Thanks

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 68 guests