Back up is a must: the AVSIM hack story

A press release, a few months ago struck my attention. AVSIM, one of the leading Flight Simulation sites, that was operating since 1996, issued the following press release:

“We regret to inform the flight simulation community that on Tuesday, May 12, AVSIM was hacked and effectively destroyed. The method of the hack makes recovery difficult, if not impossible, to recover from. Both servers, that is the library / email and web site / forum servers were attacked. AVSIM is totally offline at this time and we expect to be so for some time to come. We are not able to predict when we will be back online, if we can come back at all. We will post more news as we are able to in the coming days and weeks….”.

Actually, I was not struck by the hack itself, since it is quite obvious that the more a website is very well known, the more the possibilities that it becomes a valuable target for a hacker. In this particular case, the attacker did not perform a typical defacing (did not change the layout and contents of the portal), nor caused a Denial-of-Server (thus preventing legitimate users to access the site), but “simply” deleted the partitions of both AVSIM servers. I don’t know how the attacker performed the attack. However, I’m pretty sure he followed the usual “procedure”: initially, he exploited a vulnerability of the Web Application to gain access to the server, then he uploaded some code on the web server to make a local privilege escalation gaining the rights to use any kind of command.
Anyway, what really “shocked” me is not that despite security countermeasures were in place a website was attacked, but that a serious web business was wiped off as there wasn’t any Disaster Recovery plan foreseing content back ups off-site. The data is the business for many web-based organizations and back ups are paramount for business continuity in case of attack. Not only performing a periodic backup is important. HOW you back up data can prevent loss of money and downtime of services too.
On May 13, 2009, a quote in the AVSIM temporary forum explained that they dutifully backed up their servers every day. Unfortunately, they backed up the servers BETWEEN servers. “That is, GREEN, our library server, would be backed up to PURPLE, our WEB/Forum server. That way, if one or the other failed, we would have a back up on the remaining active server. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation”. Although cross back up is better than no-back ups and backing up on the same servers, it is not a procedure I could expect in a large organisation that makes money with its website! A proper back up for such a portal would require at least an off site back up and possibly a back up of the back up. There are various methods to make a back up and to keep it available. One could be using a mix of protected RAID (Redundant Array of Independent Disks) architecture in mirroring, striping or parity configuration and an off-site weekly back up.
For sure AVSIM had not a certified Information Security Management System (ISMS), “that part of the overall management system, based on a business risk approach, to establish, implement, operate, monitor, review, maintain and improve information security” (where Information Security means “preservation of confidentiality, integrity and availability of information”). ISO 27001 would have asked for a back up policy “to maintain the integrity and availability of information and information processing facilities” (A.10.5) and for a Business Continuity Management “to counteract interruptions to business activities and to protect critical business processes from the effects of major failures of information systems or disasters and to ensure their timely resumption” (A.14).
By the way, with the support of the community (even financial one), AVSIM was patiently restored….I hope with some lessons learned for the future!