Thread scheduling memory leak

[brendan7]

Has anybody ever had memory problems with this class?
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue

Recently I started getting a memory leak whenever I update my extension without restarting SFS. In eclipse MAT I find that class taking up 250MB of memory after about 12 hours. I am not sure what changes could have started this. It is probably related to my uses of the TaskScheduler but I believe everything is done according to best practices. I believe every task is being destroyed in the extension's destroy() method.

[brendan7]

Just realized that "destroy()" wasn't being called when the extension was automatically reloaded. I see in some docs that manually reloading is recommended for production environments. I switched it to MANUAL and will update with results.

[Lapo]

Hi,
a couple of quick suggestions:

Mmake sure you always get a reference to the tasks you're starting.
Example:

Code: Select all

ScheduledFuture<?> task = scheduler.schedule(...);

When you're done with the task or the Extension gets destroyed you will need to call

Code: Select all

task.cancel(true);

for every active task you're running.

Just realized that "destroy()" wasn't being called when the extension was automatically reloaded. I see in some docs that manually reloading is recommended for production environments. I switched it to MANUAL and will update with results.

This doesn't sound right.
destroy() is always called on the old Extension, regardless of the reloading modality, unless there's an error loading/initing the new Extension. If this is the case the former Extension is kept active and the destroy() method is not invoked.

Check your server side logs after reloading the extension. It will probably tell you what's wrong.

Cheers

[brendan7]

Thanks Lapo,

Yes, I am getting references to tasks, like so

Code: Select all

ScheduledFuture<?> startupTaskHandle;
ScheduledFuture<?> userUpdateTaskHandle;
public void scheduleTasks(){
         try{
             startupTaskHandle = sfs.getTaskScheduler().scheduleAtFixedRate(new StartupTaskRunner(), 1, 999, TimeUnit.SECONDS);
         }catch(Exception e){this.logException(e);}

         try{
             userUpdateTaskHandle= sfs.getTaskScheduler().scheduleAtFixedRate(new UserUpdateTaskRunner(), 1, 60, TimeUnit.MINUTES);
         }catch(Exception e){this.logException(e);}
}

Then cancelling after one run like so:

Code: Select all

    protected class StartupTaskRunner implements Runnable
    {
        public void run()
        {
            try{
                executeStartupTasks();
            }catch (Exception e){
                logException(e);
            }
            try{
                if(startupTaskHandle!=null)startupTaskHandle.cancel(true);
            }catch (Exception e){logException(e);}
        }
    }

Or cancelling recurring tasks in destroy() like so:

Code: Select all

    @Override
    public void destroy()
    {
        try {if (userUpdateTaskHandle!= null) userUpdateTaskHandle.cancel(true);}catch(Exception e){this.logException(e);}
    }

This doesn't sound right.

Exactly, it doesn't sound right, it is suspicious. The extension is logging a lot of information, and when an automatic extension reload occurs in the production extension, I only see init() in the logs, no destroy(). The same extension code on my test server works fine and shows destroy(), with no leak.

Is automatic reload not recommended because there are some issues? Or are you just recommending people manually reload production jars to make sure they don't make mistakes? Does it have to do with file size or number of tasks? My extension is 2.3MB and registers about 20 scheduled tasks with the TaskScheduler.

I haven't determined yet if manual reload fixes the issue.

Sometimes I see this error upon an automatic reload, although not often, so I don't think it is causing the leak. Not sure why it happens though:

Code: Select all

com.smartfoxserver.v2.exceptions.SFSExtensionException:
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Exception: com.smartfoxserver.v2.exceptions.SFSExtensionException
Message: Class not found: com.myclass.extension.MainExtension
Description: An error occurred while reloading extension: myExtension in { Zone: MyZone }
The new extension might not function properly.
+--- --- ---+
Stack Trace:
+--- --- ---+
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.createJavaExtension(SFSExtensionManager.java:358)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.createExtension(SFSExtensionManager.java:260)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager.reloadExtension(SFSExtensionManager.java:534)
com.smartfoxserver.v2.entities.managers.SFSExtensionManager$ExtensionFileChangeListener.fileChanged(SFSExtensionManager.java:111)
org.apache.commons.vfs.events.ChangedEvent.notify(ChangedEvent.java:34)
org.apache.commons.vfs.provider.AbstractFileSystem.fireEvent(AbstractFileSystem.java:519)
org.apache.commons.vfs.provider.AbstractFileSystem.fireFileChanged(AbstractFileSystem.java:487)
org.apache.commons.vfs.impl.DefaultFileMonitor$FileMonitorAgent.check(DefaultFileMonitor.java:647)
org.apache.commons.vfs.impl.DefaultFileMonitor$FileMonitorAgent.access$200(DefaultFileMonitor.java:418)
org.apache.commons.vfs.impl.DefaultFileMonitor.run(DefaultFileMonitor.java:373)
java.lang.Thread.run(Unknown Source)

[Lapo]

Is automatic reload not recommended because there are some issues?

Yes there can be issues in production but not for the kind of problem you're looking for. The problem is with uploading files from remote. Since uploading can take time, the new Extension jar file gets overwritten, thus triggering a reload before the whole transfer is complete thus causing a failure in loading the new code.

This is the reason why we recommend to use MANUAL mode instead. This way you can transfer the whole file and then trigger the reload after the jar(s) have been transferred correctly.

The fact that a destroy doesn't trigger seems utterly strange. Are you sure you're logging any messages in the destroy?

My extension is 2.3MB and registers about 20 scheduled tasks with the TaskScheduler.

No this isn't a problem.

My suggestion, if possible, is to install another copy (in another folder) of SFS2X on your production server, without touching the main one. Give it a different TCP port, maybe turn off the webserver(*), and use it for testing. Upload your current Extension there and, with no traffic, you can play with it and run some tests to see if there's any problem with reloading.

(*) = You can turn off the webserver so you have one less port to configure and you can still manage it via the AdminTool of the main SFS2X instance.

At this point I am a puzzled at what the problem might be.

Let us know