Hi
This is not directly SmartFox related problem but When garbage collector starts working it works more than 30 seconds which is big, huge, enormous problem. Now I hate garbage collector what can i do. I tried lots of vm options . Code can be optimized but even when one or two seconds stops. it is still problem
Thanks
Heap Problem
Re: Heap Problem
Hi,
a 30 seconds pause sounds extreme. How did you measure this? And how do you know it is the garbage collector?
In all of our massive load tests (even in excess of 120K CCU) we have never seen such thing. I'd need more details to understand what is really going on. I could suspect that you may have a memory leak type problem which causes long GC pauses but before that I need evidence that this is really happening.
Thanks
a 30 seconds pause sounds extreme. How did you measure this? And how do you know it is the garbage collector?
In all of our massive load tests (even in excess of 120K CCU) we have never seen such thing. I'd need more details to understand what is really going on. I could suspect that you may have a memory leak type problem which causes long GC pauses but before that I need evidence that this is really happening.
Thanks
Re: Heap Problem
Hi,
When I started smartfox server, used heap goes to 750 mb and drops to 45 mb I watch with jvisualvm goes up and down . one day later it goes to 1000 mb and drops to 250 mb like. one week later Size and Max becomes equal . now it flactuates between 2000mb and 1500 mb. in a weekly period users disconnects all of them. I look at the logs there is like 40 seconds no log. and then people disconnects. There is another instance of smartfox(community edition ) we use for testint new games. it works normally at the same time. I am thinking of gc' s stop the world thing.
How can i find memory leak?
Thanks
When I started smartfox server, used heap goes to 750 mb and drops to 45 mb I watch with jvisualvm goes up and down . one day later it goes to 1000 mb and drops to 250 mb like. one week later Size and Max becomes equal . now it flactuates between 2000mb and 1500 mb. in a weekly period users disconnects all of them. I look at the logs there is like 40 seconds no log. and then people disconnects. There is another instance of smartfox(community edition ) we use for testint new games. it works normally at the same time. I am thinking of gc' s stop the world thing.
How can i find memory leak?
Thanks
Re: Heap Problem
Fluctuations of heap are pretty normal, although a clean SmartFoxServer (without running Extensions) will not spike at 750MB on start and then drop to 45...
It is possible that your Extension code is holding onto data that is increasing the heap demand, but without details I can only speculate.
This could be many things, not just GC. For example you could have threading issues.
The first thing to do is monitoring the server's vital parameters via the admin tool, including the Queue status.
If the problem where all people is kicked out happens again take a thread dump and send it to us (email or here):
See here --> viewtopic.php?f=18&t=14458
http://docs2x.smartfoxserver.com/Advanc ... ive-server
cheers
It is possible that your Extension code is holding onto data that is increasing the heap demand, but without details I can only speculate.
I look at the logs there is like 40 seconds no log. and then people disconnects.
This could be many things, not just GC. For example you could have threading issues.
The first thing to do is monitoring the server's vital parameters via the admin tool, including the Queue status.
If the problem where all people is kicked out happens again take a thread dump and send it to us (email or here):
See here --> viewtopic.php?f=18&t=14458
How can i find memory leak?
http://docs2x.smartfoxserver.com/Advanc ... ive-server
cheers
Re: Heap Problem
Heap dump says something like this
313 instances of "com.smartfoxserver.v2.entities.SFSUser", loaded by "sun.misc.Launcher$AppClassLoader @ 0x80000048" occupy 4.959.728 (19,64%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system class loader>"
Keywords
sun.misc.Launcher$AppClassLoader @ 0x80000048
java.util.concurrent.ConcurrentHashMap$Node[]
com.smartfoxserver.v2.entities.SFSUser
Details »
Problem Suspect 2
4.448instances of "java.lang.Class", loaded by "<system class loader>" occupy 4.317.416 (17,10%) bytes.
Biggest instances:
•class org.python.core.PySystemState @ 0x804fc9d0 - 2.270.784 (8,99%) bytes.
Keywords
java.lang.Class
Details »
Problem Suspect 3
8.659 instances of "java.lang.ref.Finalizer", loaded by "<system class loader>" occupy 3.215.664 (12,74%) bytes.
Keywords
java.lang.ref.Finalizer
Which seems like japanese to me.
when there is 350 users, in around 3 minutes periods , it fluctuates between 600 mb and 40 mb.is it normal?
I will also send the heapdump to support.
Thanks.
313 instances of "com.smartfoxserver.v2.entities.SFSUser", loaded by "sun.misc.Launcher$AppClassLoader @ 0x80000048" occupy 4.959.728 (19,64%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system class loader>"
Keywords
sun.misc.Launcher$AppClassLoader @ 0x80000048
java.util.concurrent.ConcurrentHashMap$Node[]
com.smartfoxserver.v2.entities.SFSUser
Details »
Problem Suspect 2
4.448instances of "java.lang.Class", loaded by "<system class loader>" occupy 4.317.416 (17,10%) bytes.
Biggest instances:
•class org.python.core.PySystemState @ 0x804fc9d0 - 2.270.784 (8,99%) bytes.
Keywords
java.lang.Class
Details »
Problem Suspect 3
8.659 instances of "java.lang.ref.Finalizer", loaded by "<system class loader>" occupy 3.215.664 (12,74%) bytes.
Keywords
java.lang.ref.Finalizer
Which seems like japanese to me.
when there is 350 users, in around 3 minutes periods , it fluctuates between 600 mb and 40 mb.is it normal?
I will also send the heapdump to support.
Thanks.
Re: Heap Problem
Yeah... no this is not relevant.
Did you see the links I posted in my previous reply?
I was referring to a thread dump, not an heap dump. And I would recommend to do it only if an when the server manifests the behavior you have described (all users being kicked out). If that happens, don't restart the server immediately: take the thread dump first and also see if you can connect with the AdminTool.
If you can take also a couple of screenshots of the Dashboard. Both tabs: the main one and the "queue status" one.
cheers
Did you see the links I posted in my previous reply?
I was referring to a thread dump, not an heap dump. And I would recommend to do it only if an when the server manifests the behavior you have described (all users being kicked out). If that happens, don't restart the server immediately: take the thread dump first and also see if you can connect with the AdminTool.
If you can take also a couple of screenshots of the Dashboard. Both tabs: the main one and the "queue status" one.
cheers
Re: Heap Problem
Thanks,
I will do that but it happens once in a week sometimes more than a week.. it happens usually at midnight I ll do that next day if it doesnt matter when to take thread dump.
I will do that but it happens once in a week sometimes more than a week.. it happens usually at midnight I ll do that next day if it doesnt matter when to take thread dump.
Re: Heap Problem
No, it doesn't matter what time you do it.
Supposing this is a memory leak I would recommend taking a look at the Server's Dashboard every day and watching the trend of the memory growth, especially the "Allocated memory" which in the graph is the cyan / light blue one.
If you notice that there's a growing trend over several days this may indicate that you're amassing data in memory that can't be garbage collected.
On the other hand if the crash you've described is due to a "out of memory" error you should definitely see traces of it in the logs. Right around the time of the user drop.
Cheers
Supposing this is a memory leak I would recommend taking a look at the Server's Dashboard every day and watching the trend of the memory growth, especially the "Allocated memory" which in the graph is the cyan / light blue one.
If you notice that there's a growing trend over several days this may indicate that you're amassing data in memory that can't be garbage collected.
On the other hand if the crash you've described is due to a "out of memory" error you should definitely see traces of it in the logs. Right around the time of the user drop.
Cheers
Re: Heap Problem
Somthing comes to my mind I wonder if this code might cause a thread problem
robotTaskScheduler= sfs.getTaskScheduler().schedule(robot, 0, TimeUnit.SECONDS);
And robot has run method
I keep thread for every table there are about more than 200 possible robot. Could this cause to freeze smartfox so it couldnt write logs.
Thanks
robotTaskScheduler= sfs.getTaskScheduler().schedule(robot, 0, TimeUnit.SECONDS);
And robot has run method
Code: Select all
@Override
public void run() {
while(isActive) {
Player nextPlayer = table.getPlayerByIndex(table.getNextPlayer());
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
if(nextPlayer.isAvailable()){// this means that table is empty robot can play instead
if(pauseForNewCommer){
pauseForNewCommer = false;
try {
Thread.sleep(3000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
if(pause)
continue;
if(nextPlayer.isAvailable()){
// If player doesn't have 15 stone pick the stone and then throw
// Else just throw
if(!nextPlayer.hasPickedTheStone()) {
robotPickStone(nextPlayer);
}else {
robotThrowStone(nextPlayer);
}
}
}
}
extension.trace( "Robot running " + Thread.currentThread().getName());
}
I keep thread for every table there are about more than 200 possible robot. Could this cause to freeze smartfox so it couldnt write logs.
Thanks
Re: Heap Problem
Yes, you should never call Thread.sleep inside a scheduled task. Actually you should never use it inside any SmartFoxServer code!
This will block the thread for a long time causing problems to its backing threadpool.
In other words: don't block threads in order to delay the execution of some code, instead use the Task Scheduler to schedule the execution of the task in the future.
cheers
This will block the thread for a long time causing problems to its backing threadpool.
In other words: don't block threads in order to delay the execution of some code, instead use the Task Scheduler to schedule the execution of the task in the future.
cheers
Who is online
Users browsing this forum: btaletpovl and 59 guests