Heap Problem

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

User avatar
ekrem5353
Posts: 118
Joined: 10 Dec 2015, 15:50

Heap Problem

Postby ekrem5353 » 31 May 2016, 12:31

Hi

This is not directly SmartFox related problem but When garbage collector starts working it works more than 30 seconds which is big, huge, enormous problem. Now I hate garbage collector what can i do. I tried lots of vm options . Code can be optimized but even when one or two seconds stops. it is still problem

Thanks
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Heap Problem

Postby Lapo » 31 May 2016, 13:54

Hi,
a 30 seconds pause sounds extreme. How did you measure this? And how do you know it is the garbage collector?

In all of our massive load tests (even in excess of 120K CCU) we have never seen such thing. I'd need more details to understand what is really going on. I could suspect that you may have a memory leak type problem which causes long GC pauses but before that I need evidence that this is really happening.

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
User avatar
ekrem5353
Posts: 118
Joined: 10 Dec 2015, 15:50

Re: Heap Problem

Postby ekrem5353 » 31 May 2016, 14:14

Hi,

When I started smartfox server, used heap goes to 750 mb and drops to 45 mb I watch with jvisualvm goes up and down . one day later it goes to 1000 mb and drops to 250 mb like. one week later Size and Max becomes equal . now it flactuates between 2000mb and 1500 mb. in a weekly period users disconnects all of them. I look at the logs there is like 40 seconds no log. and then people disconnects. There is another instance of smartfox(community edition ) we use for testint new games. it works normally at the same time. I am thinking of gc' s stop the world thing.

How can i find memory leak?

Thanks
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Heap Problem

Postby Lapo » 31 May 2016, 14:47

Fluctuations of heap are pretty normal, although a clean SmartFoxServer (without running Extensions) will not spike at 750MB on start and then drop to 45...

It is possible that your Extension code is holding onto data that is increasing the heap demand, but without details I can only speculate.
I look at the logs there is like 40 seconds no log. and then people disconnects.

This could be many things, not just GC. For example you could have threading issues.

The first thing to do is monitoring the server's vital parameters via the admin tool, including the Queue status.
If the problem where all people is kicked out happens again take a thread dump and send it to us (email or here):
See here --> viewtopic.php?f=18&t=14458

How can i find memory leak?

http://docs2x.smartfoxserver.com/Advanc ... ive-server

cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
ekrem5353
Posts: 118
Joined: 10 Dec 2015, 15:50

Re: Heap Problem

Postby ekrem5353 » 01 Jun 2016, 07:27

Heap dump says something like this



313 instances of "com.smartfoxserver.v2.entities.SFSUser", loaded by "sun.misc.Launcher$AppClassLoader @ 0x80000048" occupy 4.959.728 (19,64%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system class loader>"

Keywords
sun.misc.Launcher$AppClassLoader @ 0x80000048
java.util.concurrent.ConcurrentHashMap$Node[]
com.smartfoxserver.v2.entities.SFSUser
Details »
Problem Suspect 2
4.448instances of "java.lang.Class", loaded by "<system class loader>" occupy 4.317.416 (17,10%) bytes.

Biggest instances:
•class org.python.core.PySystemState @ 0x804fc9d0 - 2.270.784 (8,99%) bytes.

Keywords
java.lang.Class
Details »
Problem Suspect 3

8.659 instances of "java.lang.ref.Finalizer", loaded by "<system class loader>" occupy 3.215.664 (12,74%) bytes.

Keywords
java.lang.ref.Finalizer


Which seems like japanese to me.
when there is 350 users, in around 3 minutes periods , it fluctuates between 600 mb and 40 mb.is it normal?
I will also send the heapdump to support.

Thanks.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Heap Problem

Postby Lapo » 01 Jun 2016, 07:48

Yeah... no this is not relevant.

Did you see the links I posted in my previous reply?
I was referring to a thread dump, not an heap dump. And I would recommend to do it only if an when the server manifests the behavior you have described (all users being kicked out). If that happens, don't restart the server immediately: take the thread dump first and also see if you can connect with the AdminTool.

If you can take also a couple of screenshots of the Dashboard. Both tabs: the main one and the "queue status" one.

cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
ekrem5353
Posts: 118
Joined: 10 Dec 2015, 15:50

Re: Heap Problem

Postby ekrem5353 » 01 Jun 2016, 08:19

Thanks,

I will do that but it happens once in a week sometimes more than a week.. it happens usually at midnight I ll do that next day if it doesnt matter when to take thread dump.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Heap Problem

Postby Lapo » 01 Jun 2016, 14:39

No, it doesn't matter what time you do it.
Supposing this is a memory leak I would recommend taking a look at the Server's Dashboard every day and watching the trend of the memory growth, especially the "Allocated memory" which in the graph is the cyan / light blue one.

If you notice that there's a growing trend over several days this may indicate that you're amassing data in memory that can't be garbage collected.

On the other hand if the crash you've described is due to a "out of memory" error you should definitely see traces of it in the logs. Right around the time of the user drop.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
ekrem5353
Posts: 118
Joined: 10 Dec 2015, 15:50

Re: Heap Problem

Postby ekrem5353 » 03 Jun 2016, 18:41

Somthing comes to my mind I wonder if this code might cause a thread problem

robotTaskScheduler= sfs.getTaskScheduler().schedule(robot, 0, TimeUnit.SECONDS);


And robot has run method

Code: Select all

    @Override
    public void run() {

        while(isActive) {
            Player nextPlayer = table.getPlayerByIndex(table.getNextPlayer());
            try {
                Thread.sleep(1000L);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }

            if(nextPlayer.isAvailable()){// this means that table is empty robot can play instead
                if(pauseForNewCommer){
                    pauseForNewCommer = false;
                    try {
                        Thread.sleep(3000L);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
                if(pause)
                    continue;

                if(nextPlayer.isAvailable()){
                    // If player doesn't have 15 stone pick the stone and then throw
                    // Else just throw
                    if(!nextPlayer.hasPickedTheStone()) {
                        robotPickStone(nextPlayer);
                    }else {
                        robotThrowStone(nextPlayer);
                    }
                }
            }
        }

        extension.trace( "Robot running " + Thread.currentThread().getName());
    }

I keep thread for every table there are about more than 200 possible robot. Could this cause to freeze smartfox so it couldnt write logs.


Thanks
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Heap Problem

Postby Lapo » 04 Jun 2016, 07:07

Yes, you should never call Thread.sleep inside a scheduled task. Actually you should never use it inside any SmartFoxServer code! :)
This will block the thread for a long time causing problems to its backing threadpool.

In other words: don't block threads in order to delay the execution of some code, instead use the Task Scheduler to schedule the execution of the task in the future.

cheers
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: btaletpovl and 59 guests