Page 1 of 1

error during accept():Too many open files, ulimit fails

Posted: 18 Jun 2010, 23:27
by jimmie
This is the error that I get when we receive too many connections (~500):

Code: Select all

[ WARNING ] > error during accept(): java.io.IOException: Too many open files

I see this problem has been encountered many times before, so we've followed the suggested procedure EXACTLY as it is documented here:

http://www.smartfoxserver.com/docs/docPages/config/linux.htm

Unfortunately ulimit makes no difference. We are running SmartFox as the user 'smartfox'. I'd imagine that the sfs and start.sh scripts are also being run as this user. If they're not, would this be the problem?

The client I wrote has a heartbeat. Every 30s, we ping SmartFox using roundTripBench() from ActionScript. If we're not connected, we try to reconnect instead.

Could I be DDoS'ing our SmartFox server with such code?

When we throttle our incoming connections, the server stays up and running just fine. When we re-allow all IPs to connect, it dies almost immediately.

Is there anything I could be missing? Are there any suggestions on how to debug this problem?

Any advice would be greatly appreciated, thank you.

-jimmie

Posted: 19 Jun 2010, 07:20
by Lapo
The client I wrote has a heartbeat. Every 30s, we ping SmartFox using roundTripBench() from ActionScript. If we're not connected, we try to reconnect instead.

This is not going to work. The roundTripBench is not a real "sser interaction" it only measures the lag. Since it's unlikely that a Users sends these requests manually it doesn't count as a user action.

To perform a heartbeat you should send an "empty" request (no params) to your server extension.
Could I be DDoS'ing our SmartFox server with such code?

I hope not :)

As regards the "ulimit" setting you must make sure that you run the command and the SmartFox in the same session. If you change the ulimit value, then switch user or close the terminal you will loose the settings.

Posted: 21 Jun 2010, 18:26
by jimmie
Thanks Lapo,

We solved the 'Too many open files' issue. It was in fact governed by the user's ulimit settings. Our server was apparently not configured correctly to allow the ulimit to be modified by the smartfox user. We were receiving the following error:

Code: Select all

ulimit: open files: cannot modify limit: Operation not permitted

Of course, this was coming from the sfs and start.sh scripts which were being run by some other service, so it took us a while to notice.

We followed the instructions from http://www.chrissearle.org/blog/technical/increasing_max_number_open_files_glassfish_user_debian and were able to successfully change the ulimit - everything is working great now.

Thanks for the information on roundTripBench, I'll send an empty request to our extension instead.