IT Disaster Recovery: It’s an investment of thought, not cash

Disasters happen whether we have made preparations for them or not. Even when braced, the unthinkable occurs and the situation escalates beyond even the most competent and well thought plans. One doesn’t have to look back more than a few weeks to see a good example of this; Hurricane Sandy slammed into the New York/New Jersey area, causing damage on a scale that hasn’t been seen in over a century. While you have limited options against the forces of nature, there are several key areas that you can fortify your infrastructure and minimize downtime.

Secure your website

While the idea of a criminal mastermind breaking into a home is the kind of juicy story news outlets love, the truth is that the vast majority of security breaches are caused by lack of diligence. An unlocked window, a spare key in an obvious location or even lack of lights at night are all things that increase the likelihood of a crime of opportunity. The same goes for the security of your website. Simple steps can reduce your exposure to scrutiny by leaps and bounds, and cost little to nothing to implement! Here are some examples:

• Make sure your server does not respond to pings. Hackers deploy scripts that automatically send out feelers to IP blocks to compile a list of reachable servers. If your server does not respond, you’re not added to the list, thus reducing your chance of an anonymous attack.

• Use a firewall, and use it well. This is your proverbial bouncer at the club. You may want to err on the side of caution with a whitelist (where only approved addresses and services can access your server) or a blacklist (so that anything on the list is denied access). Either way, you will automatically deflect many automated attacks and attempts by the inexperienced to connect to your resources. And this is the majority of the threat against you. Bonus points: Firewalls also log anything that is filtered through it, so even if you are compromised, there is a paper trail.

• Databases are for local use ONLY. I cannot stress this enough. Sometimes architectural considerations dictate that the database and web services should reside on different servers. But in all other cases, they should reside on the same server (behind the same firewall!). Furthermore, the database should only accept connections from the local machine. Finally, disable the root user and DO NOT copy credentials used anywhere else in your infrastructure. All of this combined limits access to required resources only.

• Disable/Remove public access to Administrative tools. Disable phpMyAdmin. Obscure access to your site’s administrative console. Prune, restrict and reset administrator credentials once every year. Audit your code and logs so you can remove any unnecessary calls from the code to the outside world, and outside access to your code.

Strength through redundancy

In the modern age, it’s not unreasonable to assume a user owns a smartphone, a computer, a web-capable console or media player, even a SmartTV. We receive data not only over traditional internet lines, but through broadband wireless providers and 3G/4G mobile networks. Even if several of these devices and/or delivery methods fail, we would still have access to the internet. This is redundancy in its purest form. And it’s a design practice that should be implemented for your production environment.

Providing redundant web services is relatively simple; all that is required is two or more web servers that provide content and a reverse-proxy server. At the most basic level, you can set up a smaller web server with a proxy service installed that will filter all your traffic to your main web server, and only redirect to your smaller server when the main site is non-responsive. This method is cheap and effective for small sites. There is a bright side to what otherwise would’ve been just another operational cost: the word redundancy goes hand in hand with the term load-balancing. With a minimum of two web servers and an independent proxy server, your web presence can scale as traffic to your site increases; spin up a new web server, add it to your proxy’s configuration and you’re done! Best of all, this system remains stable as you scale.

The only caveat to redundancy is that you MUST make sure that all of your servers automatically sync data and code, either with each other or through a central repository.

Commit to backups and a rapid recovery plan

Of all the advice given, this is probably the part that is put into practice the least. Backups cost money, and bandwidth, and provide no immediate benefit. Recovery plans are drafted and then set aside as daily issues take precedence. Projects stream in, businesses become complacent and the whole thing is forgotten. I cannot stress enough the importance of having resources on standby and a well thought-out recovery plan. Regardless of all the preparations you have made to insulate from incursions and failures, a disaster will strike.

Having said all this, however, the cost does not have to be, and should not be, high. If you host at a datacenter, or in-house, only data that is essential to your business and cannot be lost should be backed up offsite. If you use the cloud, however, it’s a simple matter to create a script that backs up your virtual hard drive and stores it for safekeeping.

Keep the following things in mind when planning for disaster recovery:

• Keep backups in a separate location than the main service.

• Use auto-reporting tools that will alert you immediately when something has gone wrong. Good examples of system monitoring tools are Nagios and Cacti.

• Have a server elsewhere ready to spin up in case of emergency (Redundancy, redundancy, redundancy!)

• Write up a plan with all essential access information included, and keep a hard copy! Also, keep digital copies in several different locations.

Securing your servers costs nothing but resource time. Committing to redundancy will increase your costs by one to four hundred dollars a month, but also offers scalability in addition to failover capabilities. And cloud backup services are incredibly cheap, some as cheap as $10 a month! When you consider whether you should invest $200/month to these recommendations, ponder this question: How much does it cost you for even one hour of downtime a month? Does that amount exceed the cost of being prepared? Don’t forget that your web presence is your public face, and even a small amount of downtime can have repercussions on your reputation.

DOOR3 is based in the heart of downtown Manhattan. We were very lucky and suffered no damage, but some of our employees were not so lucky. To them, our colleagues, clients, their families and all our neighbors affected by Hurricane Sandy, we wish for your speedy recovery.