SOLVED Questions about PermGem, ClassLoaders, RoomExtensions

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

karfau
Posts: 25
Joined: 02 Sep 2011, 17:28

SOLVED Questions about PermGem, ClassLoaders, RoomExtensions

Postby karfau » 01 Aug 2012, 17:03

First some details:
We are running a "app" where we use groovy for some of the code.
groovy, mysql-connector, and a lib with "transfer-classes" (for typed client-server-communication) is deployed in __lib__
One jar with one Zone and two RoomExtensions (and all the implementations for request- and event-handlers) is deployed to the extension-folder.
The rooms are created dynamically as games and get on of the RoomExtensions attached to them.

When monitoring the jvm we see that the PermGem is running full pretty fast (from 1 hour to some hours depends on how fast Rooms are created), until the server doesn't take any requests anymore.
therefore we tuned the instance to run with a lot more PermGemSpace (270MB currently) and activated GarbageCollection for PermGemSpace:

Code: Select all

-XX:MaxPermSize=256m
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled


Now it lasts about 10 hours before we have to restart the instance.

Through some digging we think it could be because of a new ClassLoader that loads the RoomExtension each time a room is created.
(And because some things clearly don't get cleaned up even whit garbagecollection)

Question:
- Would it be a solution to deploy Code for the dynamically created Rooms to __lib__, would this stop it from being "reloaded"?
- Is there someone with experience using groovy in combination with sfs2x? over here http://groovy.329449.n5.nabble.com/Anot ... 89594.html
it seems like there could be a problem with "SoftReferences keeping a pointer to Classes" which we will try to handle with another jvm-start-parameter:

Code: Select all

-XX:SoftRefLRUPolicyMSPerMB=150

Some Ideas about it?
-we are recognizing a lot of errors inside "BlueBox.do" when monitoring the app with AppDynamics all with those details:

Code: Select all

org.mortbay.jetty.RetryRequest : null
   at org.mortbay.jetty.nio.SelectChannelConnector$RetryContinuation.suspend(SelectChannelConnector.java:459)
   at com.smartfoxserver.v2.bluebox.BlueBox.handlePollingRequest(BlueBox.java:221)
   at com.smartfoxserver.v2.bluebox.BlueBox.doPost(BlueBox.java:121)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
   at com.smartfoxserver.v2.bluebox.SessionFilter.doFilter(SessionFilter.java:141)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
   at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   at org.mortbay.jetty.Server.handle(Server.java:326)
   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
   at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
   at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

should we worry or why shouldn't we?

I will gladly provide more details if needed,
looking forward for answers :)
Last edited by karfau on 09 Aug 2012, 08:28, edited 1 time in total.
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Questions about PermGem, ClassLoaders, RoomExtensions

Postby Lapo » 02 Aug 2012, 17:38

Hello,
Through some digging we think it could be because of a new ClassLoader that loads the RoomExtension each time a room is created.
(And because some things clearly don't get cleaned up even whit garbagecollection)

Well yes, it's no mystery, we have it documented here:
http://docs2x.smartfoxserver.com/Advanc ... assLoading

Permgen increases as new Rooms with Extension are created, which in turn will instantiate new versions of the Extension classes. For this reason is highly important that you deploy only the strictly necessary code in the extensions/ folder and the rest (i.e. dependencies) should go under extensions/__lib__/

There's even a more radical approach which is to deploy everything to the extensions/__lib__/ folder with the advantage of no more PermGen boundless growth. The downside is that extension hot-reload won't work and a restart will be needed.

- Is there someone with experience using groovy in combination with sfs2x? over here http://groovy.329449.n5.nabble.com/Anot ... 89594.html
it seems like there could be a problem with "SoftReferences keeping a pointer to Classes" which we will try to handle with another jvm-start-parameter:
Hmm, that would be bad because it could cause the issue you are facing, in other words old Classes are not free to be GCed. :(
Personally I am not able to help about groovy.

Code: Select all

org.mortbay.jetty.RetryRequest : null
   at org.mortbay.jetty.nio.SelectChannelConnector$RetryContinuation.suspend(SelectChannelConnector.java:459)
   at com.smartfoxserver.v2.bluebox.BlueBox.handlePollingRequest(BlueBox.java:221)
   at com.smartfoxserver.v2.bluebox.BlueBox.doPost(BlueBox.java:121)

Should we worry or why shouldn't we?

This is the first time I see this. The problem seems to be happening at the Jetty level when the HTTP request is suspended for a while. I would suspect that you have (or have had) a few clients whose browser/connection did not support HTTP continuations.
It can happen when clients are behind proxies or firewalls with lots of restrictions, although I haven't seen it before.

Can you reproduce the problem testing from your offices or home connection? I wouldn't expect that.

Hope it helps
Lapo
--
gotoAndPlay()
...addicted to flash games
karfau
Posts: 25
Joined: 02 Sep 2011, 17:28

Re: Questions about PermGem, ClassLoaders, RoomExtensions

Postby karfau » 02 Aug 2012, 18:28

About poutting everything under __lib__: If I do it with the Zoneextension it is not started because I can't select the class in the adminpanel.
as we only have one Zone and we can't fix the Problem with groovy, We will try the approach with deploying the RoomExtensions to __lib__.

I read through the classloading document a lot of times, but by reading it I didn't understand that creating a dynamic Room from a ZoneExtension that is contained in the same jar as the ZoneExtension will start a new ClassLoader. Maybe it would be good to use this example in the docs to make this more clear.

By the way: with the third flag we reached a runtime about 18 hours ending at 31k classes loaded and 90k unloaded.

I hope, the __lib__-approach will finally let us get rid of the problem.

thx for the answer
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Questions about PermGem, ClassLoaders, RoomExtensions

Postby Lapo » 02 Aug 2012, 19:04

About poutting everything under __lib__: If I do it with the Zoneextension it is not started because I can't select the class in the adminpanel.
as we only have one Zone and we can't fix the Problem with groovy, We will try the approach with deploying the RoomExtensions to __lib__.

Yes, deploying a Zone extension to __lib__ doesn't offer any advantages. Since Zone aren't created dynamically at runtime there's no problem. The problem is only with continuous creation/destruction of Rooms and relative Ext. code. Hope that was clear.

I read through the classloading document a lot of times, but by reading it I didn't understand that creating a dynamic Room from a ZoneExtension that is contained in the same jar as the ZoneExtension will start a new ClassLoader. Maybe it would be good to use this example in the docs to make this more clear.

Good point. We'll try to make it clearer. In essence every new Extension that is started comes with its own Class Loader to provide insulation and re-deployability. From this simple rule it should be clear that Zone Extensions are unlikely to flood the PermGen unless you keep reloading them every minute while, with dynamic Rooms the problem exists.
Normally, in Java the above problem shouldn't manifest because the PermGen can be garbage collected.
As regards Groovy, I am not familiar enough with it and if it keeps references to the classes then you will have a problem.

I hope, the __lib__-approach will finally let us get rid of the problem.

Absolutely, because this way the Extension classes are "loaded" from the parent ClassLoader which is never re-created.

Hope it's clear.
Lapo

--

gotoAndPlay()

...addicted to flash games
karfau
Posts: 25
Joined: 02 Sep 2011, 17:28

SOLVED: Questions about PermGem, ClassLoaders, RoomExtension

Postby karfau » 09 Aug 2012, 08:25

We finally are free of our problems with crashing smartfox-servers :)

Please make this more clear in the docs, especially the part that with each dynamically created Room there also is a classloader if an extension is added and that all classes are added again unless they are in the __lib__-Folder.
(And it could be even mre helpfull to make an extra hint, that if groovy or alike tools are used which create classes dynamically or keep references to classes they "enhance" this is getting even worse.)
This will save a lot of trouble I think.

Thx for the answers
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: SOLVED Questions about PermGem, ClassLoaders, RoomExtens

Postby Lapo » 09 Aug 2012, 13:43

Noted, thanks
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 54 guests