Automatic shutdown

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

keshab-lsg
Posts: 6
Joined: 09 Oct 2018, 03:46

Automatic shutdown

Postby keshab-lsg » 09 Oct 2018, 04:30

We are facing an issue of auto-shutdown of sfs server. I am adding more info below.

Server is ubuntu 18.0.4 lts, hosted in aws ec2.
Server type is c5.2xlarge having 8 cores, 16G ram.
Using Community Edition v2.13 with v2.13.3 patch.
Client is unity.
No java installed in os.

We have only 1 zone installed in sfs server, max 10 users at a time and we have added bots to simulate live playing environment. Server is getting stopped automatically atleast 2-3 times in 24 hours and there is no error or trace in the logs. We have searched inside SmartFoxServer_2X/SFS2X/logs/, SmartFoxServer_2X/SFS2X/logs/boot/, SmartFoxServer_2X/SFS2X/logs/http/ and our app named directory inside logs.

Most of the times we find INFO [Thread-0] managers.SFSZoneManager - BuddyList saveAll... in the logs before getting WARN [SFS2X ShutdownHook] core.SFSShutdownHook - SFS2X is shutting down. The process may take a few seconds..., sometime first line after second.

We have installed mysql in the same server and left it unused to verify if os is stopping processes but mysql was not stopped. Also we are running daemon processes of our own in the server and those are also not getting stopped.

We were thinking that it could be memory issue at first when we were running it in a 1G ram box but now in this server it is starting with 3605M, below is from boot.log

Processor(s): 8
VM Max. memory: 3605MB
os.name: Linux
os.arch: amd64
os.version: 4.15.0-1021-aws
java.version: 1.8.0_102
java.vendor: Oracle Corporation
java.vendor.url: http://java.oracle.com/
java.vm.specification.version: 1.8
java.vm.version: 25.102-b14
java.vm.vendor: Oracle Corporation
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.io.tmpdir: /tmp
Default charset: UTF-8


We have also checked for any system exit in our codebase.

Can you help us with debugging the issue? We can provide any required data. Also, is there a way to trace the caller of SFSShutdownHook?
User avatar
Lapo
Site Admin
Posts: 23009
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Automatic shutdown

Postby Lapo » 09 Oct 2018, 08:46

Hi,
usually this is due to the OS killing a process. I think you should check your OS logs (typically /var/log/syslog) at the time of the server shutdown.

Can you help us with debugging the issue? We can provide any required data. Also, is there a way to trace the caller of SFSShutdownHook?

The JVM shutdown hook simply responds to an OS signal (typically SIGTERM), so a JVM stack trace is not sufficient to determine the true origin of the shutdown.

As a side note, I find it strange that this is happening under Ubuntu in AWS as we've been working in the same environment and OS lately and have never seen any problem.

In any case let us know what you find in the logs.
Lapo
--
gotoAndPlay()
...addicted to flash games
keshab-lsg
Posts: 6
Joined: 09 Oct 2018, 03:46

Re: Automatic shutdown

Postby keshab-lsg » 09 Oct 2018, 09:15

Hi Lapo,

Checked syslog, there are no incident logged 12 mins prior and 9 mins after the incident.

Below if from sfs log:
08 Oct 2018 | 21:00:41,055 | INFO | Thread-0 | entities.managers.SFSZoneManager | | BuddyList saveAll...
08 Oct 2018 | 21:00:41,055 | WARN | SFS2X ShutdownHook | v2.core.SFSShutdownHook | | SFS2X is shutting down. The process may take a few seconds...
08 Oct 2018 | 21:00:41,056 | INFO | Thread-1 | entities.managers.SFSBannedUserManager | | BanUser data saved.


Below is from syslog:
Oct 8 20:48:36 ip- systemd-timesyncd[538]: Network configuration changed, trying to establish connection.
Oct 8 20:48:36 ip- systemd-timesyncd[538]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Oct 8 21:09:01 ip- CRON[18759]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 8 21:09:18 ip- systemd[1]: Starting Clean php session files...
Oct 8 21:09:19 ip- systemd[1]: Started Clean php session files.


(Edited to remove the ip)

We are actually out of debugging options, any help will be appreciated.
User avatar
Lapo
Site Admin
Posts: 23009
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Automatic shutdown

Postby Lapo » 09 Oct 2018, 10:11

Interesting (and very odd) that there are no signs in the syslog.

Can we exclude more mundane explanations, such as someone from your team just logging in the Admin and either inadvertently or willingly shutting down the server?
We had a case, years ago, where a client was seeing strange server behaviors then he realized he hadn't changed the standard Admin user/pass and someone had logged in and messed with the config.

Other than that we could either take a look ourselves if you can share an SSH access to the machine, or we could setup an SFS2X on Ubuntu 18 on EC2 and see if it's reproducible.

As I mentioned earlier, we have worked quite a lot with EC2 and Ubuntu recently and have had no problems. The only difference could be the OS version, as most of our work has been done under Ubuntu 16 rather than 18, though I can't imagine how Ubuntu itself could interfere.

In any case let us know. If you decide for the SSH access, you can get in touch with us via our support@... email. Just add a link to this discussion for reference.

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Lapo
Site Admin
Posts: 23009
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Automatic shutdown

Postby Lapo » 09 Oct 2018, 10:11

Interesting (and very odd) that there are no signs in the syslog.

Can we exclude more mundane explanations, such as someone from your team just logging in the Admin and either inadvertently or willingly shutting down the server?
We had a case, years ago, where a client was seeing strange server behaviors then he realized he hadn't changed the standard Admin user/pass and someone had logged in and messed with the config.

Other than that we could either take a look ourselves if you can share an SSH access to the machine, or we could setup an SFS2X on Ubuntu 18 on EC2 and see if it's reproducible.

As I mentioned earlier, we have worked quite a lot with EC2 and Ubuntu recently and have had no problems. The only difference could be the OS version, as most of our work has been done under Ubuntu 16 rather than 18, though I can't imagine how Ubuntu itself could interfere.

In any case let us know. If you decide for the SSH access, you can get in touch with us via our support@... email. Just add a link to this discussion for reference.

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
keshab-lsg
Posts: 6
Joined: 09 Oct 2018, 03:46

Re: Automatic shutdown

Postby keshab-lsg » 09 Oct 2018, 10:18

I can assure that access is with 2 people and we are the guys scratching our head :D

This is our prod grade server which has our apis, cronjobs and stuff, it will be a bit problematic to share as we have to undo all setup and redo later. If possible plz try to spawn an instance from your side.

Also, just did a dmesg, there is no oom-killer in the response.
User avatar
Lapo
Site Admin
Posts: 23009
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Automatic shutdown

Postby Lapo » 09 Oct 2018, 10:27

keshab-lsg wrote:I can assure that access is with 2 people and we are the guys scratching our head :D

Yeah, I understand. I just needed to exclude the more trivial issues :)

This is our prod grade server which has our apis, cronjobs and stuff, it will be a bit problematic to share as we have to undo all setup and redo later. If possible plz try to spawn an instance from your side.

Yes we'll do.
I would also recommend a test that you could run on your side:
Install another copy of SFS2X in a different folder and start it up (while the other SFS2X is not running) with the default configuration. Let it run for 24 hours and see if it does the same.

If it doesn't it might indicate something wrong with the first SFS2X installation.

Let us know.
Lapo

--

gotoAndPlay()

...addicted to flash games
keshab-lsg
Posts: 6
Joined: 09 Oct 2018, 03:46

Re: Automatic shutdown

Postby keshab-lsg » 09 Oct 2018, 10:32

Thanks Lepo, will try spawning a fresh copy and turn off existing one.
User avatar
Lapo
Site Admin
Posts: 23009
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Automatic shutdown

Postby Lapo » 09 Oct 2018, 15:49

Ok, let us know how it goes.
If it behaves in the same way let us know the exact name of the Ubuntu image you're using so we can try to setup a similar environment.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
keshab-lsg
Posts: 6
Joined: 09 Oct 2018, 03:46

Re: Automatic shutdown

Postby keshab-lsg » 10 Oct 2018, 03:41

Hi Lepo,

We have deployed a fresh sfs copy with the patch in another server and it has been 12 hours since, it is running.
As next step we are going to put our jar and xml (zone config) files now and keep it running without any connections for another 12-24 hours. If this works fine, then we will start connecting monitor further. Will keep you posted.

I missed to give some info about the extra libraries we are depending upon, are:
gson 2.8.2
httpclient 4.5.2
httpcore 4.4.4
memcached-java-client 3.0.2
mysql-connector-java 5.1.23

Do you think there might be any compatibility issue with above jars?
User avatar
Lapo
Site Admin
Posts: 23009
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Automatic shutdown

Postby Lapo » 10 Oct 2018, 07:55

Thanks,
I missed to give some info about the extra libraries we are depending upon, are:
gson 2.8.2
httpclient 4.5.2
httpcore 4.4.4
memcached-java-client 3.0.2
mysql-connector-java 5.1.23

I don't see how any of these libs would cause the application to shutdown with no errors.
By the way, the two httpclient / httpcore jars you've listed are the same version of those shipped with SFS2X already.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
keshab-lsg
Posts: 6
Joined: 09 Oct 2018, 03:46

Re: Automatic shutdown

Postby keshab-lsg » 10 Oct 2018, 08:05

Thanks Lepo, it's me being over-cautious.

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 55 guests