[Solved]Strange SFS problem

Need help with SmartFoxServer? You didn't find an answer in our documentation? Please, post your questions here!

Moderators: Lapo, Bax

HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

[Solved]Strange SFS problem

Postby HCBMedia » 21 Jan 2010, 14:25

I have 2 SmartFox servers running. One running on IP .40 (develop version) and one running on IP .41 (live version).
I have bought a license for it.
The version I am using is 1.6.6.
There are 12 zones (4 different games, 3 versions per game).

I will try to describe the problem. Note: this has happened 2 times since November last year (2009).

The situation is as follows. The SFS (live) server runs perfectly for quite a lot of days. Than suddenly, all the people that want to play the game, cannot connect anymore.
When I try to connect using the AdminTool, it keeps disconnecting (I believe it's a connection refused or timed out error). I keep on trying and after a while I am logged in.
The mainpage of the admintool (normally) shows some statistics/counters. When this problem occurs, the counters are all set to 0 and the loading bar (at the bottom) keeps loading, even when I switch to another tab (which actually doesn't succeed).

I have talked with our hosting and our flash-developing-company, but they don't know.

There is nothing to see in the logs, just users disconnecting.
The only solution there is, is using the "./sfs restart" command on the server (admintool was kinda broken).

Has anyone of you ever experienced such a thing and perhaps found a solution? It's a big issue for me/us (my company), since playing these games is something that is used a lot. If games are not available, it gives a lot of (angry) questions and it's not a good service.

So this is kind of my last option/hope. If someone wants the (wrapper)log, I can send it, but not much to see there.


PS: I'm dutch, so I hope you understand my little story :p
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 22 Jan 2010, 07:31

There is nothing to see in the logs, just users disconnecting.

Really? not even a warning? nothing?
Can you double check?

Are you running your own extensions?
If so are you sure they are working correctly. Again I would expect you are also logging some info from your code.

Are you monitoring the server during its activity?
We would suggest to do it both from its Admin Tool and from the external OS resource monitor (CPU/Ram usage)

Also when this problem happens you could attempt to obtain a thread dump in the JVM.
The thread dump in the console can be done by hitting ctrl+break (Windows) or sending kill -SIGQUIT to the process (under Linux/Unix)

Additionally if you run Java6 you could use a visual tool such as VisualVM to analyze what's going on:

https://visualvm.dev.java.net
https://visualvm.dev.java.net/description.html
Lapo
--
gotoAndPlay()
...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 22 Jan 2010, 07:51

I am running "own" extensions, but they are not developed by me (external company). So I do not have much influence on that part.

The thing I do when such a problem occurs is login to the server itself (the machine) and watch some processes (check ./sfs status says it's running). Top also doesn't tell me much.

The log(s) I am referring to are "wrapper_yyyymmdd.log". Are there any other logs I should/could check?

I want to attach the logfile as attachment, but I don't know how. Is it possible?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 22 Jan 2010, 07:58

I am running "own" extensions, but they are not developed by me (external company). So I do not have much influence on that part.

Custom code can do all sorts of nasty things if not done properly so I would definitely recommend to monitor what's going on there.

A thread dump is absolutely needed without any other info.
Alternatively one or more VisualVM snapshots could help as well.
Monitoring the amount of threads in use via the AdminTool is also a little help.

As for the logs you can zip them and send them to our email.
Click Support > Contact Us from the main menu in this page
Lapo

--

gotoAndPlay()

...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 22 Jan 2010, 08:02

How can I get information/debug info from those extensions? And how to get a thread dump? I'm running the SFS server on a unix machine.

The problem is: if such a problem occurs, I cannot get any information from the AdminTool, so getting the amount of threads will than be 0 or nothing.
Currently, the amount of threads is 30, with 19 users connected.

Does that VisualVM work on unix?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 22 Jan 2010, 08:19

How can I get information/debug info from those extensions?

Normally, developers log warnings and debug information to the standard server log to keep track of requests, problems, unexpected states etc...
And how to get a thread dump? I'm running the SFS server on a unix machine.

Please re-read my 1st post it is explained there.

The problem is: if such a problem occurs, I cannot get any information from the AdminTool, so getting the amount of threads will than be 0 or nothing.

Yes, but I am saying that you should monitor the machine and Server continuously not just when the problem appears. By knowing what has happened before we can start to understand the possible causes of the issue.

The two likely candidates here are an OutOfMemory exception and/or a thread problem. Thread usage can be easily monitored from the AdminTool.
Keeping an eye on the CPU and RAM usage of the Server process is also good, as well as to monitor the machine in general.

Does that VisualVM work on unix?

Absolutely
Lapo

--

gotoAndPlay()

...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 22 Jan 2010, 08:43

For the record: I only have access to a unix machine (and no gui). I guess something visual doesn't work than.

Is there some way to get the threads from someplace besides admintool? I don't want to keep an eye on the admintool every minute. So if I can retrieve those statistics using some sort of command, I could write a cronjob/log these numbers.

CPU and memory usage of the server itself (the machine) cannot be the problem (0.2% in use, 2.5GB free).
Is there some setting to adjust the memory-limit that the SFS process can/may use?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 23 Jan 2010, 11:36

Is there some way to get the threads from someplace besides admintool?

With a thread dump.
Is there some setting to adjust the memory-limit that the SFS process can/may use?

http://www.smartfoxserver.com/docs/docP ... nced.htm#6
Lapo

--

gotoAndPlay()

...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 25 Jan 2010, 09:45

zicht@fox:/$ ps u -p 14718
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
zicht 14718 0.4 4.6 2332552 378560 ? Sl Jan04 146:47 /home/zicht/SFS-dev/SFS_PRO_1.6.6/jre/bin/java -server -Dfile.encoding=UTF-8 -Djava.util.log.....
zicht@fox:/$ kill -SIGQUIT 14718
zicht@fox:/$


I have executed the command as you said. However, I do not get any output. What am I doing wrong, do I have the wrong process?

The SFS server has (already) 2.0 GB of memory allocated. I guess that should be sufficient, or should I make 3/4GB of it? The server itself has 8GB of memory.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 25 Jan 2010, 17:33

I have executed the command as you said. However, I do not get any output. What am I doing wrong, do I have the wrong process?

The output is fired in the log files.

The SFS server has (already) 2.0 GB of memory allocated. I guess that should be sufficient, or should I make 3/4GB of it? The server itself has 8GB of memory.

There's no need to go past 2Gb. (and you can't unless you run a 64-bit JVM)
Just adding more more memory is not a good strategy if not needed as it might cause heavier garbage collection phases.
Lapo

--

gotoAndPlay()

...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 26 Jan 2010, 07:39

Thanks. I have send an email with the output of the thread dump (zipped).
This is a thread dump that has just been made, but there are no problems. I don't know if you can do anything with it or not.

I assume that I should do a thread dump as soon as the problem occurs again, but I'd like to stay ahead of the problem.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 26 Jan 2010, 08:04

Thank you. The thread dump looks good, nothing "suspicious" in it
Lapo

--

gotoAndPlay()

...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 26 Jan 2010, 08:06

Lapo wrote:Thank you. The thread dump looks good, nothing "suspicious" in it


Ok. What else can I/we do to find the cause / prevent the problem from happening again? Or is it just wait until it happens and perform a thread dump then?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 26 Jan 2010, 08:08

Not knowing what the problem is I have no idea how to prevent it.
Please keep monitoring the server and give us more details.
Lapo

--

gotoAndPlay()

...addicted to flash games
HCBMedia
Posts: 14
Joined: 29 Oct 2009, 10:15

Postby HCBMedia » 01 Feb 2010, 08:12

And just now, the problem appeared again.
A colleague of my was already working on it. I did perform a threaddump, so I hope it was not too late.

I have send the threaddump to the emailaddress again.

This problem starts to irritate me, especially since nobody knows a solution (don't take this personally).
Is there really nothing I can do, to debug/test/investigate? I'm willing to do (almost :p) everything.

Can it have something to do with the develop-sfs server running also on that machine?

Return to “SmartFoxServer 1.x Discussions and Help”

Who is online

Users browsing this forum: No registered users and 30 guests