Page 1 of 2

Proximity Manager really big

Posted: 07 Jan 2020, 02:30
by trianglehead
Hi! I am noticing the ProximityManager bloating and eventually causes performance issues, CPU spiking, etc. I wonder if this following code causes this issue:

Code: Select all

               
               list = mroom.getProximityList(u);
               if(list!=null) {
                  list.add(u);// I think prox list doesn't include self, and we have coded everything to assume self is included
               }
               

Image
Annotation 2020-01-06 203216.jpg
(95.31 KiB) Not downloaded yet

Re: Proximity Manager really big

Posted: 07 Jan 2020, 08:21
by Lapo
Hi,
the proximity system can use quite a bit of memory as it basically maps the world in a 3D data structure that keeps track of every user.
However there are multiple variables at play, such as the number of players in an MMORoom, their distribution in space and the available hardware resources.

As regards the code snippet, I am not sure what it is and I'd need more context to understand what you're doing.

Thanks

Re: Proximity Manager really big

Posted: 15 Jan 2020, 16:13
by trianglehead
Hi. So I had to restart the server, and let the server run again as it grinded to a halt. With the same amount of user base, around 100-150 ccu all day everyday, the MMORoom object just keeps growing over time. Currently it has run for 7 days, and the object grew to 2g in size. First couple days when I did a jcmd heap dump it was only 150m.

Image


My code uses MMOROOM in this manner:
1.)

Code: Select all

 mroom.getProximityList(pos, aoi); // to get list of users at certain positions and specify an area of interest

2.) I also create MMOItems:

Code: Select all

ISFSMMOApi mmoApi = SmartFoxServer.getInstance().getAPIManager().getMMOApi();

      // Set the Item in the room at specific coordinates
      mmoApi.setMMOItemPosition(item, new Vec3D(x, y, z), room);


But if you look at the attachment, the memory consumption is with SFSUser objects. With only 100-150 ccus, after 7 days There are 145k SFSUser objects and over 2gig retained heap size.


What is it that I can do? I'm running Smartfox 2.13.6.

Re: Proximity Manager really big

Posted: 15 Jan 2020, 18:39
by Lapo
Are you doing any manual Room management such as calling MMORoom.addUser or MMORoom.removeUser?
Since the Room is apparently not releasing User objects, have you noticed any anomaly in the number of joined Users?

What are the hardware specs of the server running SFS2X?

Thanks

Re: Proximity Manager really big

Posted: 15 Jan 2020, 19:01
by trianglehead
Hi. Thanks for replying.

I only join room like this:

Code: Select all

getApi().joinRoom(sfUser, rooms.get(0));

Specs are EC2 c5.2xlarge with 16 gigs ram which I allocate 8 gigs, but the specs are not the issue, I got another game running on Smartfox with 7-10x the ccu. But does not use MMORoom and can run for months and no memory issues.

Any other idea would be greatly appreciated.

Re: Proximity Manager really big

Posted: 16 Jan 2020, 09:09
by Lapo
Please also let us know about my previous question:
Since the Room is apparently not releasing User objects, have you noticed any anomaly in the number of joined Users?

Thanks

Re: Proximity Manager really big

Posted: 16 Jan 2020, 11:20
by Lapo
I went back to the top of the original post and noticed this code snippet that you posted.

Code: Select all

list = mroom.getProximityList(u);
               if(list!=null) {
                  list.add(u);// I think prox list doesn't include self, and we have coded everything to assume self is included
               }


Can you explain what you're trying to accomplish with this?
When you're referring to "self" what is it? From the server-side perspective there is no "self". Because the server is not an entity in the game.

This is likely the cause of the problem, by the way, because you're not supposed to manipulate the ProximityList manually.
Thanks

Re: Proximity Manager really big

Posted: 16 Jan 2020, 13:38
by trianglehead
Thank you for reply. Self is a sfs user object of the user who made this call. I had changed this since last reboot so the original list no longer gets modified as I suspected that to be the cause. But still the bloat happens. This is how I changed it since last reboot but problem persists.

Code: Select all

list = mroom.getProximityList(u);
               if(list!=null) {
List newList = new ArrayList(list);
                 
                  newList.add(u);// I think prox list doesn't include self, and we have coded everything to assume self is included
return newList;
               }


So that original list isn't modified.

Re: Proximity Manager really big

Posted: 16 Jan 2020, 16:04
by Lapo
In order to investigate this we would need to see the server logs: ideally the last 1-2 days before the server was restarted.
Also we would need to see you server side code to better understand what you're doing.

If you're running a commercial license, you can send us the material by zipping it and emailing it to our support@... email box.
(Please add a reference to this conversation as well)

Thanks

Re: Proximity Manager really big

Posted: 08 Aug 2020, 22:36
by trianglehead
So.. I'm still having this issue. I just worked around it by changing room names everyday, but that's not a good solution for me. So to update, when the server only has 200-300ccu, after 2 days, I have 95,000 instances of SFSUsers. Digging deeper with Eclipse memory analyzer I took a screenshot(redacted some class names to stay anonymous):
Image
This screenshot is just digging deep into 1 of the 95,000 SFSUser instances and trying to find out who is keeping it from getting garbage collected. The User has already been gone for days. But looks like when an MMORoom stays alive, it keeps a reference of room owner. Even if the room owner has already left.

Now the room owner has a list of SFSUser in the proxyList, and the proxyList holds a whole bunch of SFSUsers, and each SFSUser in this list also has their own proxyList, and they never go away since the owner of the room is forever referenced by the MMORoom.
Here's a SFSUser instance that's reference by a proxyList of another user who is referenced by another proxylist, and it goes on.


Here is a better example of what I just explained. As you can see. This example I picked another random user who no longer is online and traced the reference all the way to the MMORoom, which has an owner who will forever be reference by the MMORoom, and the owner has a proxyList, each user in the proxyList all have a proxyList and so on..
Image
Here is a similar situation, not with room owner, but with proximitimanager. This might be a different thing though.
Image


Just FYI I don't manually add users, I use getApi().createRoom(getParentExtension().getParentZone(), config, sfUser) and getApi().joinRoom(sfUser, room);

Currently, after writing all of this I'm thinking maybe I have answered one of the questions myself... Perhaps I should create the room with a null user, will that solve the issue? I will try it.

Re: Proximity Manager really big

Posted: 10 Aug 2020, 07:15
by Lapo
Now the room owner has a list of SFSUser in the proxyList, and the proxyList holds a whole bunch of SFSUsers, and each SFSUser in this list also has their own proxyList, and they never go away since the owner of the room is forever referenced by the MMORoom.

Aren't these MMORooms removed at some point?

In any case creating Rooms with the owner set to null is probably a good idea, in this case.

Let us know

Re: Proximity Manager really big

Posted: 10 Aug 2020, 11:45
by trianglehead
Lapo wrote:
Now the room owner has a list of SFSUser in the proxyList, and the proxyList holds a whole bunch of SFSUsers, and each SFSUser in this list also has their own proxyList, and they never go away since the owner of the room is forever referenced by the MMORoom.

Aren't these MMORooms removed at some point?

In any case creating Rooms with the owner set to null is probably a good idea, in this case.

Let us know

The MMORooms are never supposed to be released by design because it's a persistent world. But due to the leak I had no choice but to release the rooms. That why for the past while I have been renaming the room every midnight as a work around. But that is not very desirable. Every midnight when people leave the room, and come back they will join a new room, while the players in the old room who don't leave will remain in the old room until they leave and come back. Further more, MMOItems in the old room will be lost in the perspective of the players who moved to the new room. It just very workaround-ish.

Does Smartfox not recommend MMORooms that remain forever?

Re: Proximity Manager really big

Posted: 10 Aug 2020, 12:37
by Lapo
If MMORooms are supposed to linger for a long time it's best to that they are owned by the server itself (owner == null), rather than a User for the reasons you have outlined.

Does Smartfox not recommend MMORooms that remain forever?

There are many use cases, but I think the solution is the above (make sure owner is the Server).
As regards MMORooms remaining forever, it depends. MMORooms are heavier compared to regular Rooms due to the mechanisms that keep track of all Users and Items.
Since MMORooms can host 1000s of players a single server shouldn't normally run hundreds or thousands of MMORooms. If you need that many MMORooms it will probably hit memory usage, but it still should be feasible given enough resources.

Cheers

Re: Proximity Manager really big

Posted: 11 Aug 2020, 21:42
by trianglehead
Lapo wrote:If MMORooms are supposed to linger for a long time it's best to that they are owned by the server itself (owner == null), rather than a User for the reasons you have outlined.

Does Smartfox not recommend MMORooms that remain forever?

There are many use cases, but I think the solution is the above (make sure owner is the Server).
As regards MMORooms remaining forever, it depends. MMORooms are heavier compared to regular Rooms due to the mechanisms that keep track of all Users and Items.
Since MMORooms can host 1000s of players a single server shouldn't normally run hundreds or thousands of MMORooms. If you need that many MMORooms it will probably hit memory usage, but it still should be feasible given enough resources.

Cheers

So not putting an owner when creating the MMORoom only partially reduce the SFSUser counts. After 2 days, I did a thread dump and I still get 63,000 instances of SFSUser adding up to 1gig ram in the proximity manager of an MMORoom. Unless this is intended, which I don't see why since there are never 63000 CCUs, perhaps Smartfox has a memory leak with MMORooms? Perhaps you can verify the dump if you message me I can send you the link.

Re: Proximity Manager really big

Posted: 12 Aug 2020, 09:02
by Lapo
We're not aware of memory leaks with MMORooms, but maybe it depends on the context and the specifics in which they are used.
It's difficult to say.

We can try to reproduce the problem but we would need more details to understand the general approach of your game.
You explained that you create a number of MMORooms that are never removed. So is it an ever growing number? Is there a limit?
How many users would normally join a Room, on average?

Thanks