Problem after updating SSL certificates

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

grookier
Posts: 26
Joined: 28 Sep 2016, 10:40

Problem after updating SSL certificates

Postby grookier » 26 Mar 2020, 15:11

Good evening,
a few months ago I enabled HTTPS on smartfoxserver2 following the official documentation http://docs2x.smartfoxserver.com/GettingStarted/cryptography

Everything worked correctly without ever having to intervene.
Today I connected to the admin address and the certificate is unsafe, and my apps no longer work.

I followed the guide again, I gave the same commands to create the new keystore.jks file, I imported it from the admin panel, but the certificate was still insecure by connecting to https://my_domain.it:8443/admin/.

So I restarted the server. Now the panel is no longer accessible in any way, however it seems that smartfoxserver starts correctly from the logs.
console log
https://pastebin.com/K4wr5mET

smartfox.log
https://pastebin.com/qX8qP36d

boot.log
https://pastebin.com/Kra2Mbb7

server.xml

Code: Select all

  <Connector SSLEnabled="true" clientAuth="false" keystoreFile="lib/apache-tomcat/conf/keystore.jks" keystorePass="***********" maxThreads="200" port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true" sslProtocol="TLS"/>



If I try to connect to the admin panel, the connection times out ERR_TIMED_OUT

EDIT
Without making further restarts for 15 minutes, suddenly it worked and the certificate is safe.

What is this slowness due to?

Is there any way to automatically update the keystore.jks file, or do I have to create a script?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Problem after updating SSL certificates

Postby Lapo » 26 Mar 2020, 16:53

Hi,
for starters it's not clear why the certificate was no longer valid. Did it expire? If so did you renew it?

As regards the delay you mentioned at the end of the post:
in your smartfox.log file the last log message available is

Code: Select all

SmartFoxServer 2X (2.14.0) READY!

This is actually not the last boot message as a few seconds later you should also see this message:

Code: Select all

BlueBox Service (4.0.0) READY.


This is what it looks like on my laptop:

Code: Select all

17:50:08,687 INFO  [SFSWorker:Sys:1] v2.SmartFoxServer     - SmartFoxServer 2X (2.15.0) READY!
17:50:11,317 INFO  [TomcatRunner] bluebox.BBSessionFilter     - BlueBox Service (4.0.0) READY.

It takes roughly 2-3 extra seconds for Tomcat to boot and initialize the SFS2X-related services.

If you attempt to connect via HTTP before Tomcat is ready you will likely get an error. Maybe on your server Tomcat is booting up very slowly? Check your log files and see how long it takes before the BlueBox message appears.

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
grookier
Posts: 26
Joined: 28 Sep 2016, 10:40

Re: Problem after updating SSL certificates

Postby grookier » 27 Mar 2020, 11:05

Hello Lapo,

Yes, the certificates had expired, I automatically update those of the domain with lets'encrypt, but apparently I have to automatically regenerate them and re-import them for smartfox. So yes, I regenerated and improtated them.

This is the delay that I recovered from yesterday's log:

Code: Select all

26 Mar 2020 | 16:17:44,049 | INFO  | SFSWorker:Sys:1 | smartfoxserver.v2.SmartFoxServer |     | SmartFoxServer 2X (2.14.0) READY!
26 Mar 2020 | 16:26:01,136 | INFO  | main | tomcat.bluebox.BBSessionFilter |     | BlueBox Service (4.0.0) READY.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Problem after updating SSL certificates

Postby Lapo » 27 Mar 2020, 16:21

Hi,
that looks pretty weird! :shock:
Unless SFS2X is running on a Commodore C64 you should not be seeing 9 minutes between those two events :)

Jokes aside, it seems very strange. Is it possible your server machine was busy doing something else when you booted up SFS2X?
What are the hardware specs of the machine?

Thanks
Lapo

--

gotoAndPlay()

...addicted to flash games
grookier
Posts: 26
Joined: 28 Sep 2016, 10:40

Re: Problem after updating SSL certificates

Postby grookier » 27 Mar 2020, 18:01

These are the server specs, it doesn't seem so bad to me :D :

CPU:
4 vCore
RAM:
8 GiB
Storage:
100 GiB

I have nothing in use besides smartfoxserver, it is a server dedicated only to this.

I also restarted the server during the various tests.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Problem after updating SSL certificates

Postby Lapo » 28 Mar 2020, 11:58

Well, there's no indication what CPU is being used.
Since it's a virtualized CPU you're getting a slice of the actual hardware and from the boot times you're getting it looks like a very small slice.

As a comparison I've tried running SFS2X 2.14 on the smallest Amazon EC2 instance, a t3.nano.
This is what it looks like:

Code: Select all

28 Mar 2020 | 11:54:03,033 | INFO  | SFSWorker:Sys:1 | smartfoxserver.v2.SmartFoxServer |     | SmartFoxServer 2X (2.14.0) READY!
28 Mar 2020 | 11:55:00,431 | INFO  | main | tomcat.bluebox.BBSessionFilter |     | BlueBox Service (4.0.0) READY.


Also keep in mind that the boot process is entirely mono-threaded, so multiple cores won't make a difference. Still I find it hard to believe that Tomcat requires 9 minutes to boot on your server.
I would recommend checking with a CPU monitor what happens during the SFS2X boot and if all CPU is used only by its process or if it's contended among other services. If 9 minutes is the best you can get it might be an indication that these kind of virtual servers are pretty bad :(

Hope it helps
Lapo

--

gotoAndPlay()

...addicted to flash games
grookier
Posts: 26
Joined: 28 Sep 2016, 10:40

Re: Problem after updating SSL certificates

Postby grookier » 28 Mar 2020, 18:18

Thanks for the info, I will be monitoring the CPU status at the next reboot.
User avatar
holyfiregames
Posts: 19
Joined: 16 Jul 2019, 19:11

Re: Problem after updating SSL certificates

Postby holyfiregames » 05 Jul 2020, 23:01

Hi, I'm seeing this same issue. After my SSL certificate expired and I've re-setup everything I am having the same connection issues where I can't connect after a reboot. I'm letting it sit now as I'm not getting the bluebox message either to see if it shows up after 10 minutes like the other user here. Any idea what might cause this?

ubuntu-s-2vcpu-4gb-nyc3
4 GB Memory / 80 GB Disk / Ubuntu 16.04.5 x64

Are my server stats.

Is there anything that might cause tomcat to hang? Any logs I can check?

Thanks
User avatar
holyfiregames
Posts: 19
Joined: 16 Jul 2019, 19:11

Re: Problem after updating SSL certificates

Postby holyfiregames » 05 Jul 2020, 23:03

Here's how long it took to work:

22:53:39,248 INFO [SFSWorker:Sys:1] v2.SmartFoxServer - SmartFoxServer 2X (2.16.0) READY!
22:55:39,252 INFO [pool-1-thread-1] stats.CCULoggerTask - CCU stats: { Zone: PeskyUndead }, CCU: 0/0
22:55:39,253 INFO [pool-1-thread-1] stats.CCULoggerTask - CCU stats: { Zone: --=={{{ AdminZone }}}==-- }, CCU: 0/0
22:55:39,253 INFO [pool-1-thread-1] stats.CCULoggerTask - CCU stats: { Zone: endless-defender-idle-td }, CCU: 0/0
22:55:39,253 INFO [pool-1-thread-1] stats.CCULoggerTask - CCU stats: { Zone: BasicExamples }, CCU: 0/0
22:55:39,253 INFO [pool-1-thread-1] stats.CCULoggerTask - CCU stats: { Zone: GazpoIO }, CCU: 0/0
22:55:39,253 INFO [pool-1-thread-1] stats.CCULoggerTask - CCU stats: CCU: 0/0
05-Jul-2020 23:03:04.840 WARNING [TomcatRunner] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [563,527] milliseconds.
23:03:04,846 INFO [TomcatRunner] bluebox.BBSessionFilter - BlueBox Service (4.0.1) READY.

---------------------------
Any idea what this is: 05-Jul-2020 23:03:04.840 WARNING [TomcatRunner] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [563,527] milliseconds.

It appears this is the slow down?
User avatar
holyfiregames
Posts: 19
Joined: 16 Jul 2019, 19:11

Re: Problem after updating SSL certificates

Postby holyfiregames » 05 Jul 2020, 23:06

I did a google search and came across this: https://blog.longyb.com/2019/06/09/tomc ... d_english/ I'm not sure how to edit any of this though.
User avatar
holyfiregames
Posts: 19
Joined: 16 Jul 2019, 19:11

Re: Problem after updating SSL certificates

Postby holyfiregames » 05 Jul 2020, 23:16

In this folder: /home/SFS/SmartFoxServer_2X/jre/lib/security

File - java.security

I searched for file:/dev/random and replaced it with file:/dev/./urandom

It fixed my problem right away, do you see any security issues with this?

23:16:37,320 INFO [SFSWorker:Sys:1] v2.SmartFoxServer - SmartFoxServer 2X (2.16.0) READY!
23:16:39,254 INFO [TomcatRunner] bluebox.BBSessionFilter - BlueBox Service (4.0.1) READY.

It now loads in just a few seconds.

Thanks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Problem after updating SSL certificates

Postby Lapo » 06 Jul 2020, 07:55

Hi,
thanks for researching this. We have never seen this issue before, even in production where we mostly use different versions of Debian or Ubuntu.
In any case I think the article explains pretty well what the problem is.

It fixed my problem right away, do you see any security issues with this?

In all honesty I cannot say :) I am not a cryptography expert.
As per the article you have linked, it doesn't seem very likely that an attack will come into existence:

    "While /dev/urandom is still intended as a pseudorandom number generator suitable for most cryptographic purposes, the authors of the corresponding man page note that, theoretically, there may exist an as-yet-unpublished attack on the algorithm used by /dev/urandom, and that users concerned about such an attack should use /dev/random instead.[4] However such an attack is unlikely to come into existence, because once the entropy pool is unpredictable it doesn’t leak security by a reduced number of bits."
Lapo

--

gotoAndPlay()

...addicted to flash games
grookier
Posts: 26
Joined: 28 Sep 2016, 10:40

Re: Problem after updating SSL certificates

Postby grookier » 27 Jul 2020, 08:50

Hi, I only read it now, I still experience the slowness problem. I will try to inform myself better about this modification that you have made and eventually I will try to solve as you suggested

@Lapo I redo a a question asked in the first post of the thread:

Is there any way to automatically update the keystore.jks file, or do I have to create a script?

let me explain better, after the ssl certificates are updated with certbot, is there a way to automatically generate the new keystore.jks file and replace it with the existing one? or do I always have to execute all the commands suggested in the official guide?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Problem after updating SSL certificates

Postby Lapo » 27 Jul 2020, 09:40

Hi,
grookier wrote:let me explain better, after the ssl certificates are updated with certbot, is there a way to automatically generate the new keystore.jks file and replace it with the existing one? or do I always have to execute all the commands suggested in the official guide?

yes, every time you renew the SSL certificate you will have to do the steps in the guide.
It may be possible to use a script to automate most of the process.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 44 guests