AdminTool Restart Fails - Possible Workaround?

[Void*]

Unable to restart the server using the Admin Tool. Shuts down but fails to come back online.

THIS SEEMS TO ONLY OCCUR IF YOU CHANGE THE SFSADMIN PASSWORD BEFORE DOING ANYTHING ELSE

Setting the admin password back to the original fails to rectify the issue.

I also tried re-writing the entire config directory but that also didn't fix the issue.
I replaced the entire folder with a fresh copy of SFS2X 2.17.0 which then allowed me to restart the server normall.
As soon as I changed the sfsadmin password the problem came back.

So I reset everything again and this time I created an additional admin account with the same password I was previoiusly changing the sfsadmin account to.
Server restarts continued to work

I then deleted the sfsadmin account
Server restarts continued to work

I reset back to fresh to reproduce the issue.
I changed a couple of server settings (such as debug mode) and saved; restarts continued to work.
I changed the sfsadmin password and saved; restarts continued to work.

I reset everything again.
This time, the first action in the Admin Tool I made was to change the sfsadmin account password. Server restarts failed

There is nothing in the logs to indicate any kind of failure at any point (other than the standard socket shutdown logging). fwiw I'm running 2.17.0 on Amazon Linux 2.

Given that I have a workaround I'm not urgent for a fix, but I hope this might be able to point you in the right direction to fix it, or at least reproduce it.

[Lapo]

Hi,
could you explain the steps better? We are not able to reproduce the problem.

I have tested under Ubuntu 20:
- installed SFS2X 2.17
- logged in as the default sfsadmin user with the default password
- in the AdminTool changed the sfsadmin password and hit "Submit"
- restarted the server
- after ~10 seconds, logged in again as sfsadmin with the new password

No issues there.
Let us know.

p.s. = also can you reproduce it anywhere else? Such as on your local machine?

[Void*]

Hi Lapo,

I seem to be able to reproduce this consistently. Both on my local machine (WSL 2 running Ubuntu 18.10) and on a couple of different AWS VM instances (Amazon Linux 2 and Ubuntu 18.10).

It appears to be after making changes to the Server Configuration it just stops restarting. Beyond that there's no further information I can get for you. I can, however, reproduce it for you on demand using our instances.

Things I've tried in the meantime (because it's stopped working again after I changed a server config var);
- Rolling back the config (doesn't work)
- Replacing the various data files with originals (doesn't work)
- Deleting datafiles that aren't shipped with the SFS2X download (doesn't work)
- Replacing the SFS2X binaries in isolation (doesn't work)
- Refreshing the entire folder with a newly downloaded copy, then copying in my existing configuration and extensions (works)

For what it's worth, the java process disappears and never comes back. It looks like it might be crashing on shutdown but doesn't log why. I can't see anything in journalctl either.

If I run in standalone mode, this is what we see before it returns us to the terminal

Code: Select all

15:38:40,709 INFO  [SocketWriter::TCP-1] core.SocketWriter     - SocketWriter::TCP threadpool shutting down.
15:38:40,707 WARN  [SocketWriter::TCP-2] core.SocketWriter     - SocketWriter::TCP thread interrupted
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.smartfoxserver.bitswarm.core.SocketWriterV3$TCPWriteRunner.run(SocketWriterV3.java:818)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
15:38:40,709 INFO  [SocketWriter::TCP-2] core.SocketWriter     - SocketWriter::TCP threadpool shutting down.
15:38:40,708 WARN  [SocketReader] core.SocketReader     - Problems in SocketReader main loop: java.lang.InterruptedException: sleep interrupted, Thread: Thread[SocketReader,5,main]
15:38:40,710 WARN  [SocketReader] core.SocketReader     - java.lang.InterruptedException: sleep interrupted
        java.lang.Thread.sleep(Native Method)
        com.smartfoxserver.bitswarm.core.SocketReader.run(SocketReader.java:173)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

15:38:43,705 INFO  [SFSWorker:Ext:4] sessions.DefaultSessionManager     - Session removed: { Id: 1, Type: DEFAULT, Logged: Yes, IP: REMOVED:49393 }
15:38:44,214 INFO  [:::SFSRestarter:::] util.SFSRestart     - Process restarted: java.lang.UNIXProcess@714be048
15:38:44,723 INFO  [Thread-0] managers.SFSZoneManager     - BuddyList saveAll...
15:38:44,724 INFO  [Thread-1] managers.SFSBannedUserManager     - BanUser data saved.
15:38:44,730 WARN  [SFS2X ShutdownHook] core.SFSShutdownHook     - SFS2X is shutting down. The process may take a few seconds...
[ec2-user@ip-REMOVED SFS2X]$


[Void*]

Addition:

Could it be running out of order?

Code: Select all

15:38:44,214 INFO  [:::SFSRestarter:::] util.SFSRestart     - Process restarted: java.lang.UNIXProcess@714be048
15:38:44,723 INFO  [Thread-0] managers.SFSZoneManager     - BuddyList saveAll...
15:38:44,724 INFO  [Thread-1] managers.SFSBannedUserManager     - BanUser data saved.
15:38:44,730 WARN  [SFS2X ShutdownHook] core.SFSShutdownHook     - SFS2X is shutting down. The process may take a few seconds...


It restarts, then the shutdown hook runs.

I've also noticed this on the Extension hot reload. The hot reload spins up the new extension code, then the shutdown hook runs after the new one is online. I was using the shutdown hook to remove existing rooms between reloads, but the removal code was running after the spinup code, so it removed the *new* rooms as well

[Lapo]

Thanks,

It appears to be after making changes to the Server Configuration it just stops restarting. Beyond that there's no further information I can get for you. I can, however, reproduce it for you on demand using our instances.

Can you provide a set of steps that reliably reproduces the problem?

Cheers

[Void*]

So the below steps always caused it to fail for me but please read below the list first;

Create AWS EC2 Amazon Linux 2 t3.medium instance
Open All firewall ports to your IP
Connect to instance via SSH
Install Java 8 Runtime -

Code: Select all

sudo yum install java-1.8.0-openjdk

Issue command

Code: Select all

ulimit -c unlimited

Use wget to download SFS2X 2.17.0
Extract tar using

Code: Select all

tar xf <downloadfilename>

Enter the created SmartFoxServer_2X/SFS2X directory
Run

Code: Select all

./sfs2x-service start

Log in to SFS2X Admin Tool
Change the sfsadmin password
Hit the restart server button

I found it incredibly odd that given the number of posts on the issue that you were unable to reproduce it with what are fairly simple steps. So, I did a bit more digging into the JVM itself. It appears that the default install JVM is silently crashing (1.8.0_272-b10). When the server restarts the JVM produces a mem_align exception and shuts down.

IMPORTANT BIT
I installed the AdoptOpenJDK which is versioned at 1.8.0_282-b08 using OpenJ9 and the problem has gone. No matter what I do I cannot reproduce this issue. I have tried repeatedly, with every combination of everything that broke it before.

Installing OpenJDK 1.8.0_272-b10 immediately causes the problem to re-appear.

tldr; looks like it's not your problem really. OpenJDK 8 seems to not work properly.

[Lapo]

Hi,
thanks for the details.

Installing a JDK in the system should not affect SFS2X as it uses its own embedded JRE, regardless of what is installed in the system.
(In fact you don't need to install a JDK/JRE at all, to run SFS2X)

In order to use a different JRE you would need to manually replace it in the SmartFoxServer_2X/ installation folder (or remove it, so that the launcher searches the system's JRE)
Is that what you did, prior to starting up SFS2X?

Cheers

[Void*]

No, though logically I agree with what you're saying.

I wonder if there is some interaction going on here with the JVM dependency libraries, i.e. it uses the ones in the installed JDK path by default rather than the bundled ones. This could explain it, as afaik you guys bundle the Adopt OpenJDK with SFS2X.

The reason that the JDK is included as an install step is due to some other applications on the server, and remote build. It was easier to install the JDK than to manually set up all the JVM config/paths/dependency installs and end up "dirtying" the SFS2X JRE. At least that was the idea.

Edit: I did a test without pre-installing the JDK via YUM and it works fine. Cannot re-produce the issue

[Lapo]

We've never seen issues with a global JDK/JRE running in the system. The embedded JRE is always prioritized.
Also, we run many Linux boxes where there's a system-wide JDK installed (different version and vendor) and SFS2X running with its own JRE.

It was easier to install the JDK than to manually set up all the JVM config/paths/dependency installs and end up "dirtying" the SFS2X JRE. At least that was the idea.

Yeah, looks like it.

Very strange indeed.

[Void*]

On that note; I just changed a couple of server variables and it stopped restarting again.

No errors anywhere on the system this time. So.. I have no idea anymore.

Appears to be random; I'll stick with issuing restarts on the server itself.