Some of you might have noticed that the next.audio server was behaving erratically a few days in a row the other week. I always want to update everyone on what happened when things go wrong but I also wanted to make sure that we got to the bottom of it before posting a post-mortem. Here’s a look behind the scenes.
Where’s my memory?
I woke up one morning to the server been unavailable. Why those things always seem to happen in the middle of the night, I don’t know but I guess Murphy’s law is still going strong around here. The next.audio server handles not only the web site but also the app authentication so this is always troubling when things don’t work properly. Thankfully, the auth server being unavailable was not an immediate problem as an authorization is valid for up to 15 days after it’s been delivered and so no users would be left stranded for the perceivable future. It was preventing people from buying the app though which is annoying.
After looking at the server logs, it was clear the problem was that the server was running out of memory:
Of course this is NOT supposed to happen in regular conditions so something was afoot. The server guys determined that the server was running an ungodly amount of mail server processes, managed to kill all of them and we started looking at wat could be the root of the problem. We didn’t find anything. Next night, here we go again. Around the same time, the server dies and once again this is cause by running out of memory due to too mail server processes. Ugh.
Mail, what mail?
Now the next.audio server is not a mail server. It’s a web server. So this to begin with was suspicious. We decided to look at what mail messages were being sent and we found that those were post sharing emails, the kind that you send when you want to share a post on the site with your friends. In this case, and only in this case, is the server itself is used to send those emails. So someone was sharing a crazy amount of posts and causing the server to crash. At first this seems flattering (it was mainly my old post on Ableton EQ settings) but of course, it turned out to be a lot more malicious that that:
The info above was redacted to protect the innocent but the important part is that the emails were being sent to a bunch of email at the qq.com domain which is a chinese instant messaging service run by Tencent. If you looked at the content of the email you could see that the post sharing was used to send spam in the sharing email’s body. Great.
Clean up, clean up, everybody everywhere…
So the gig was up. Spammers had been using my server to send spam. A lot of spam. the only indication of the magnitude of this is that the number of bounced emails I got thru that period of time topped 450k. That gives you an idea of how many probably did go thru. Must be millions. Wow.
Forgetting for a moment how messed up that is and how spammers, like so many people nowadays, have no sense of morals or accountability (if you rely on scamming people and causing havoc to make a living then that says a lot about you as a person) it was time to fix things and clean it up. First order of the day was removing the share post by email feature from the website, second order was to disable any mail serving feature from the site and move any mail functionality off site so that this kind of thing never happens again.
You live, you learn
I’m not a web guy and so every little problem is an opportunity for me to learn more about it. This was not a hack, per se, and the server was never compromised. Those idiots were just using a legitimate feature of the site to their advantage. My site never made it unto spam blacklists (geez, how many emails do you need to spam in order to make it on there???) and therefore I never got a heads up that this was going on until the server started failing because of it.
Lessons learned here: 1) Be careful what you enable on the site. Everything can potentially be mis-used. 2) Keep an eye on the usage logs and try to detect odd behaviors before it gets that bad. 3) Separate any non web-serving functionality to another server in order to distribute the load. 4) Spammers are low-life losers who will never achieve anything in life.