Out of memory Permgen space

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Out of memory Permgen space

Postby Democre » 14 Dec 2010, 03:59

I am running into an out of memory error in RC1a.
The server has been up and running for only about a week.

Code: Select all

14 Dec 2010 00:11:25,509 WARN  [com.smartfoxserver.v2.controllers.ExtensionController-1] v2.controllers.ExtensionController     -
java.lang.OutOfMemoryError: PermGen space

14 Dec 2010 00:11:26,450 INFO  [SocketReader] bitswarm.sessions.DefaultSessionManager     - Session removed: { Id: 1193, Type:DEFAULT, Logged: Yes, IP: ...snipped... }

Gonna bounce it now. I would hate to have to bounce it once a week or even more frequently under heavier load... :(

Update
It appears this is caused by a lot of redeployment. Could you tell me where in the sfs2x-service script to add the following vm parameters:

Code: Select all

-XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=128m

Thanks
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 14 Dec 2010, 16:19

The problem is only related with the number of redeploys of extensions in your system. On each redeploy new class definitions are loaded into a special section of the JVM memory called PermGen (permanent generation) which holds the class bytecode.

PermGen is not part of the heap and it's not affected by the Garbage collector which makes it a potential problem for environments where class definitions continue to grow, unless you specify extra settings for the JVM as you wrote.

. Could you tell me where in the sfs2x-service script to add the following vm parameters:

Just open the file with any text editor and you'll find the directions:

# Uncomment the following line to override the JVM search sequence
# INSTALL4J_JAVA_HOME_OVERRIDE=
# Uncomment the following line to add additional VM parameters
# INSTALL4J_ADD_VM_PARAMS=
Lapo
--
gotoAndPlay()
...addicted to flash games
Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Postby Democre » 14 Dec 2010, 17:55

Thanks! Although RC1a may have a problem. I haven't done any redeploys in a week and a half, and have even restarted the server since redeployment.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 14 Dec 2010, 19:11

No this is insufficient to explain the issue.
You can run hundreds of redeploys under normal conditions. Evidently there's something else going on that we are not aware of.

I can assure you that I've run tons of redeploys without issues on dozens of different deployments :)

Anyways if you can provide a reproducible scenario I will be very happy to test it!
Lapo

--

gotoAndPlay()

...addicted to flash games
Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Postby Democre » 17 Jan 2011, 21:42

Lapo, I am encountering this problem again. We recently published our game on Kongregate.

Now every 600 - 100 connections, the server freezes with the Out of memory permgen space. The only thing that helps is a restart. We are not redeploying any classes, just restarting the server. We have restarted it about 4 or 5 times this weekend for the same error.

We are using rc1b, and the unity client.

I am out of ideas.
Alex
Posts: 131
Joined: 03 Sep 2006, 09:15

Postby Alex » 18 Jan 2011, 06:45

Do you use buddies?
Maybe its related to my problem:

viewtopic.php?t=9713
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 18 Jan 2011, 08:46

Are you really sure the OOM is due to the perm-gen memory and not the heap instead?

I'd like to see the exception details. Also I would like you to check the logs for other possible problems that might lead to that issue.
Another possibility is badly configure JVM but it's probably less likely.
Lapo

--

gotoAndPlay()

...addicted to flash games
Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Postby Democre » 18 Jan 2011, 17:02

Lapo,
I am seeing the same error as posted in my first post.

Code: Select all

v2.controllers.ExtensionController     -
java.lang.OutOfMemoryError: PermGen space

Unfortunately, the logs I have don't show the error as they have already been rolled off.

I am seeing a different error, related to logout. That is the only other error I see.

Code: Select all

java.lang.IllegalArgumentException: Logout failure. Session is not logged in: { Id: 126, Type: DEFAULT, Logged: Yes, IP: 112.xxx.xxx.xxx:33403 }
   com.smartfoxserver.v2.controllers.system.Logout.execute(Logout.java:30)
   com.smartfoxserver.v2.controllers.SystemController.processRequest(SystemController.java:127)
   com.smartfoxserver.bitswarm.controllers.AbstractController.run(AbstractController.java:96)
   java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   java.lang.Thread.run(Unknown Source)

This is because I am calling disconnect after logout. If I don't call disconnect, the client hangs and doesn't exit immediately.

In the client I sometimes get the MySQL timeout error, but that is from my use of connections(which I am positive I am closing in a finally block). I will switch to using the executeQuery with args when the bug in that gets fixed.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 19 Jan 2011, 07:26

Well,
the PermGen only contains class data. This indicates that your server code is creating too much of those. I am wondering why. You should probably give us more details about what your Extension does.

Typically new classes are created by running many class loaders, maybe you are integrating external libs in your code that causes the problem.

I would also recommend to use a tool like VisualVM or jconsole to monitor the PermGen memory at runtime and see what happens.
Another consideration is fine tuning the JVM PermGen size if your application has special requirements.
Lapo

--

gotoAndPlay()

...addicted to flash games
Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Postby Democre » 19 Jan 2011, 18:46

My extension does nothing out of the ordinary. I don't play with class loaders or dynamically creating classes. I don't modify byte code. I don't do anything except what is the normal use case for the server, which is register event and request handlers and handle them.

The only thought that I had, which I have since taken care of, is that I did have two extension directories sff and sff2 both of which had similar jars in them. One was used by one set of zones, and I had a second set of zones for testing. The jars were not exact copies of each other, but did have different versions of the same class.

The class loader should have been able to distinguish, but am wondering if there is an edge case within the smartfoxserver classloading code that doesn't handle this.

I.E.
starfall zone and starfallReg zone both pointed at sff
starfall2 zone and starfallReg2 zone both point at sff2

There is only one jar in both of those folders. This jar has same named packages and classes, just one is modified for test purposes.

The only external library I use is visural. And it was only added recently. I had the permgen error before.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 20 Jan 2011, 09:48

No, this doesn't justify the problem.
As I said there must be continuous class definitions creation in order to fill up the PermGen and cause the problem.

Monitoring via VisualVM or jconsole is essential here.

Does the problem happen in your test environment?
What kind of production server do you use? Does it have enough resources for the JVM?
Lapo

--

gotoAndPlay()

...addicted to flash games
Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Postby Democre » 20 Jan 2011, 18:27

My test environment is the same one as the production server. Just differently zoned. It is on an EC2 instance.

I hear what you're saying as far as continuous class definitions. My code does nothing of that sort. I don't use any reflection at all. I don't alter any bytecode. I don't call any compiler. I'm not even doing redeploys. I don't hold references to classes in other extension directories. It is frustrating.

I don't know how I would set up a profiler on the EC2 right now. I did try to put up a jsp which gave memory snapshots, but JSP support is not configured. I looked at the config files, and it looks like jsp support should be configured, it just isn't. So no go there either.

*EDIT* I did just try to run jconsole, but I have huge latency and it is unusable.

As for resources, according to admin tool, I've only allocated at most 30 Megs of the ~400 Meg maximum. But that is heap space. I believe the default max for permgen is set at 64 Megs.
Democre
Posts: 77
Joined: 16 Sep 2010, 17:58

Postby Democre » 20 Jan 2011, 19:42

Alright, I did run visual vm, and the permgen size starts very close to maximum (within 100k)

I have increased the permgen maximum size to 128. Maybe this should be default in the start scripts for sfs2x.
User avatar
Lapo
Site Admin
Posts: 23007
Joined: 21 Mar 2005, 09:50
Location: Italy

Postby Lapo » 21 Jan 2011, 12:48

Interesting. That explains it.
I tested on my laptop (it's a Mac) and my PermGen max size is 88MB of which only 32MB are used at startup.
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 49 guests