XML Attack, Server Issues

Need help with SmartFoxServer? You didn't find an answer in our documentation? Please, post your questions here!

Moderators: Lapo, Bax

Zelcer
Posts: 17
Joined: 26 Jan 2013, 04:46

XML Attack, Server Issues

Postby Zelcer » 01 Nov 2017, 10:23

I've recently been having some issues with my server. Before I being, here are the specs:

Server Version
SFS Pro 1.6.6 patched to 1.6.19
OS
Ubuntu Server 12.04 "Precise Pangolin" LTS
CPU
Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Cores : 12
Cache : 15360KB
RAM
4x 32GB
Disks
2 x 2000 GB
Motherboard
X10SRi-F
Kernel version
3.14.32-xxxx-grs-ipv6-64

Onto my issue. Yesterday a player spammed the server with malformed XML requests ala Billion Laughs Attack. This caused the server to overload and eventually crash. I modified some of the AntiFlood settings in the config.xml file:

Code: Select all

<AntiFlood active="true">
   <MinMsgTime tolerance="5">1000</MinMsgTime>
   <MaxRepeatedMessages>3</MaxRepeatedMessages>
   <WarningsBeforeKick>1</WarningsBeforeKick>
   <WarningMessage><![CDATA[No flooding allowed!)]]></WarningMessage>
   <KickMessage><![CDATA[You've been warned! No flooding! Now you're kicked]]></KickMessage>
   <BanMessage><![CDATA[Stop Flooding!! You're being banned]]></BanMessage>         
   <BanAfter timeSpan="50">3</BanAfter>
</AntiFlood>


After doing so, I haven't experienced the attack again, but I can't confirm it's been fixed.

Anyways, the reason I mention this is because today I've been encountering some different issues, and it seems unlikely that they're unrelated. The server started to log a bunch of "I/O Error during accept loop: Too many open files" errors, eventually causing the server to crash, or kick players randomly. This is the first time I've encountered this error, even with 1500+ concurrent users online. I've since increased the ulimit on the server, and can confirm the limit has been permanently changed, even after relogging and restarting the server. I've also increased the ulimit on the start.sh script and sfs file in the start() function. I see the correct number (currently 64,000 global, 20,000 for SFS) when running the ulimit -n command.

However, after doing so, the server will now eventually (not immediately) begin lagging, and refusing to complete the login process for users. The "top" command shows Java CPU usage at over 100%, while the Server Load is always 0% in the Admin Tool. I located the thread that was causing the issue and created a thread dump with jstack. I can provide the entire dump to you if necessary, but here's the info on that specific thread:

Code: Select all

"selector" prio=10 tid=0x00007b076c179400 nid=0x2ecc runnable [0x00007b07a890a000..0x00007b07a890ab40]
   java.lang.Thread.State: RUNNABLE
   at net.n3.nanoxml.XMLElement.findAttribute(XMLElement.java:589)
   at net.n3.nanoxml.XMLElement.hasAttribute(XMLElement.java:878)
   at net.n3.nanoxml.StdXMLBuilder.addAttribute(StdXMLBuilder.java:255)
   at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:546)
   at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:290)
   at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:630)
   at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:290)
   at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:630)
   at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:290)
   at net.n3.nanoxml.StdXMLParser.scanData(StdXMLParser.java:232)
   at net.n3.nanoxml.StdXMLParser.parse(StdXMLParser.java:203)
   at it.gotoandplay.smartfoxserver.SmartFoxServer.dispatchEvent(SmartFoxServer.java:1207)
   at it.gotoandplay.smartfoxserver.SmartFoxServer.readIncomingMessages(SmartFoxServer.java:1080)
   at it.gotoandplay.smartfoxserver.EventReader.run(EventReader.java:32)
   at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
   - None


Before the server crashes but after the lag issues begin, I profiled how long each request to the server takes. Every request, and even tasks from schedulers, take significantly longer to execute.

I'm not sure is this is because of the attacker, or because of poor server configurations, etc. I don't believe it has to do with the actual extension code, ie. an infinite loop, because I haven't modified the server code for a while and just started experiencing these issues after the XML Bomb attack.

Do you have any ideas?
User avatar
Lapo
Site Admin
Posts: 22999
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: XML Attack, Server Issues

Postby Lapo » 02 Nov 2017, 12:20

I've already replied to your PM. Let's keep the conversation in one place.
Thanks
Lapo
--
gotoAndPlay()
...addicted to flash games

Return to “SmartFoxServer 1.x Discussions and Help”

Who is online

Users browsing this forum: No registered users and 32 guests