java.lang.OutOfMemoryError: Direct buffer memory

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

java.lang.OutOfMemoryError: Direct buffer memory

Postby pratts » 16 Feb 2017, 11:51

I recently encountered the below mentioned exception while running JAVA client api to test load on smartfox server:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.<init>(SocketSendBufferPool.java:159)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool.<init>(SocketSendBufferPool.java:46)
at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:84)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.<init>(NioClientSocketPipelineSink.java:74)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:135)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:105)
at sfs2x.client.core.sockets.TCPSocketLayer.initNetty(TCPSocketLayer.java:84)
at sfs2x.client.core.sockets.TCPSocketLayer.connect(TCPSocketLayer.java:195)
at sfs2x.client.bitswarm.BitSwarmClient.connect(BitSwarmClient.java:222)
at sfs2x.client.bitswarm.BitSwarmClient.reconnect(BitSwarmClient.java:382)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketError(BitSwarmClient.java:445)
at sfs2x.client.bitswarm.BitSwarmClient.access$3(BitSwarmClient.java:441)
at sfs2x.client.bitswarm.BitSwarmClient$4.dispatch(BitSwarmClient.java:116)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnError(TCPSocketLayer.java:168)
at sfs2x.client.core.sockets.TCPSocketLayer.handleErrorThread(TCPSocketLayer.java:129)
at sfs2x.client.core.sockets.TCPSocketLayer.access$8(TCPSocketLayer.java:121)
at sfs2x.client.core.sockets.TCPSocketLayer$1.run(TCPSocketLayer.java:113)
at java.lang.Thread.run(Thread.java:745)

Upon checking the mentioned exception, I found there is a bug in netty related to memory leak [issue][/https://issues.jboss.org/browse/NETTY-424?_sscc=t].
There is also a github issue posted for the same [github-issue][/https://github.com/netty/netty/issues/1574].
Can you just check if the above issue is present in the smartfox server also because some users experienced the same issue after netty 4.0. and smartfox 2.9.2 and 2.12.1 use netty-3.9.0.Final?

JAVA client api version : 1.1.0
Smartfox server version : 2.9.2
OS : Centos 6.5
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby Lapo » 16 Feb 2017, 12:49

SmartFoxServer 2X runs it's own socket engine. We only use Netty for websockets and we have no reports or evidence of any issues on the server side.

The error you have shown is related to the client side API. Maybe you're pushing too many instances of the client in the same JVM?

Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games
pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby pratts » 16 Feb 2017, 13:28

Thank you for responding.
I am running 1000 users in a single java process with the below mentioned parameters:
java -Xmx3072M -XX:MaxPermSize=128M -Xss256k -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:-HeapDumpOnOutOfMemoryError

I tried to run 3 such processes on a system and I am using 2 systems to create the load, but somehow all the users related to a process disconnect at the same time. I tried to take the heap and thread dump but the process is unresponsive.

I read the 100K concurrent users white paper and thought we can try the same.
I am also getting the below mentioned exceptions:
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 5138
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: Unknow SFSDataType ID: -128
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:225)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSDataType. Expected: 18, found: 0
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:182)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 4608
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalStateException: Invalid SFSObject key length. Found = 4608
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:205)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeObject(DefaultSFSDataSerializer.java:729)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.decodeSFSObject(DefaultSFSDataSerializer.java:212)
at com.smartfoxserver.v2.protocol.serialization.DefaultSFSDataSerializer.binary2object(DefaultSFSDataSerializer.java:170)
at com.smartfoxserver.v2.entities.data.SFSObject.newFromBinaryData(SFSObject.java:149)
at sfs2x.client.core.SFSProtocolCodec.onPacketRead(SFSProtocolCodec.java:47)
at sfs2x.client.core.SFSIOHandler.handlePacketData(SFSIOHandler.java:281)
at sfs2x.client.core.SFSIOHandler.onDataRead(SFSIOHandler.java:130)
at sfs2x.client.bitswarm.BitSwarmClient.onSocketData(BitSwarmClient.java:428)
at sfs2x.client.bitswarm.BitSwarmClient.access$2(BitSwarmClient.java:422)
at sfs2x.client.bitswarm.BitSwarmClient$3.dispatch(BitSwarmClient.java:110)
at sfs2x.client.core.EventDispatcher.dispatchEvent(EventDispatcher.java:52)
at sfs2x.client.core.sockets.TCPSocketLayer.callOnData(TCPSocketLayer.java:162)
at sfs2x.client.core.sockets.TCPSocketLayer.handleBinaryData(TCPSocketLayer.java:156)
at sfs2x.client.core.sockets.TCPSocketLayer.access$7(TCPSocketLayer.java:155)
at sfs2x.client.core.sockets.TCPSocketLayer$NettyIOHandler.messageReceived(TCPSocketLayer.java:303)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

After 3 Hours, 14 Minutes:
Number of clients connected:7918
CPU Usage: 6.13%
Memory < 1 GB
Number of rooms: 1231
Transferred data: 33.95 GB out | 674.39 Mb in
Current Data transfer rate: OUT: 1.97 MB/s | IN: 141 KB/s
Avarage Data transfer rate: OUT: 2.91 MB/s | IN: 57.72 KB/s
Dropped Packets: Out: 448867 (0%) | In: 0

Just after this, one client process stops responding and all 1000 users disconnect. It raises the CPU and Memory on Smartfox server.

We are not sure whether this is due to large amount of data or the problem with the client code.

Do you have any idea about this. Can you share any problems faced while working on 100K users ?

Configuration of system on which smartfox server is installed:
version: 2.9.2
OS : Centos 6.5
8 cores
15 GB RAM
-Xmx4096M

Configuration of system on which clients are:
OS: Centos 6.5
8 cores
61 GB RAM
version: 1.1.0
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby Lapo » 16 Feb 2017, 14:51

Hi,
you won't be able to reach 100K users with this approach. We used a more sophisticated tool that still uses the API but replaces the socket engine with a much more light-weight one that doesn't create thousands of Netty's Channel Factories, thread pools etc... This in turn allows to keep resource consumption to a minimum, running a single Channel Factory and a single thread pool.

If you're running on Linux keep also in mind that when you approach the 1000 connection mark you will need to fine tune the "ulimit" parameter to allow for more file descriptors, as this is also a source of issues.

I tried to run 3 such processes on a system and I am using 2 systems to create the load, but somehow all the users related to a process disconnect at the same time. I tried to take the heap and thread dump but the process is unresponsive.


Try to reduce the number of clients per JVM until it runs stable. Then run one single JVM for a while (say 1 hour) and make sure there are no problems. Then you can start more JVMs until your CPU / RAM resources reach 90%, but make sure you don't max out either of those (cpu/ram) otherwise the test clients will start lagging behind and misbehave.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby pratts » 21 Feb 2017, 12:31

Thank you for the help.

I tried the method you mentioned and did not face continuous disconnection since yesterday.

I have one more question:
When the server user count crossed 8000 (approx.) users. Around 1000 users disconnected, and the CPU utilization on Smartfox panel crossed 60% (its per core I guess). The system has 8 cores, that means it is 8*60=480%.

Is this normal for smartfox ? When you tested for 100K users, did you see the same CPU spike when users got disconnected?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby Lapo » 21 Feb 2017, 14:18

Yes, massive disconnections at once will necessarily create a spike because the server has lots of work to do, clearing many data structures, sending updates to all remaining clients, resetting room and user variables etc...

Typically however, it is just a spike, so it will last for a very short time.
Lapo

--

gotoAndPlay()

...addicted to flash games
pratts
Posts: 34
Joined: 23 Aug 2016, 17:46

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby pratts » 14 Mar 2017, 08:56

when User count 21891 Users were connected, there were 54 threads in the admin panel and after sometimes when around 19118 were disconnected, thread count reached to 94 after 2-3 minutes.
Is it normal for the server ?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: java.lang.OutOfMemoryError: Direct buffer memory

Postby Lapo » 14 Mar 2017, 09:58

Yes, the number of threads oscillates based on the load.
Disconnecting large number of clients at once is a lot of work for the server, cleaning up lots of data and sending updates to everyone else.

cheers
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 61 guests