Emergency Preparedness & IT Part 1: Power

In light of the recent Hurricane Sandy events, it may be a good time to bring up the "what if" of major catastrophes.  I'm no expert in this field, but I've done my share of recovery and preparedness planning.  I'm going to present some of the things to consider and plan for, in the event that your data center & network gets seriously damaged.  This is NOT a "preppers" article, or a "SHTF" article.  This is purely focused on networks, servers, and getting them through the "tough times of down times".  This is a multi-part series, just to make it easier to read and parse...don't want to scare anyone away with one giant article..
Tragic events happen.  First and foremost you must make sure that you and your family is safe, and no one is in danger.  Once your own life is taken care of, it's time to take care of work...provided you can even get there.

First thing's first: no electricity, no point in trying.  You need to make sure your facilities have power.  In the typical environment, there are two parts/steps to this.  The first step is to cover short term power loss (a few minutes), so you must make sure that all your equipment is on a UPS system.  A few things to note:

• UPS batteries don't last forever.  They typically need to be replaced every 3-5 years.  If at 5 years the UPS doesn't say it needs new batteries yet, replace them anyway.  Put a sticker on each UPS with the date of the batteries to track this easily.
• Periodically test your UPSs by killing power to each one at least once every 6 months and making sure it stays up more then 60 seconds.
• Along the same lines, you need to test ALL UPSs at once (if you have more then one) at least once a year.  For example, if you have 4 UPSs, turn power off for all 4 at the same time, to make sure they all transition correctly.  If not, you could end up with an overload condition on one UPS (if you have servers with multiple power supplies).
• UPSs themselves don't last forever.  They are relatively simple devices, so they do last a long time, but not forever.  Make sure they all fully charge their batteries and all function OK in the tests mentioned above.
• Recycle your batteries properly.  UPS batteries are not really good for the environment, especially considering the number of batteries UPSs use in medium to large systems.  Typically junk yards and other scrap metal recycling yards will take them, and sometimes even pay you for them.

For anything longer then a minute or two, you need step two: a standby generator.  These are wired into the building or room's main power feed via an automatic transfer switch.  That will signal the generator to start and transfer power over to it in the event of a power failure.  Typically standby generators are power either by diesel, natural gas, or propane.  If you can get the service, natural gas is the best way to go.  With natural gas, you're not relying on keeping a fuel source locally (like a tank of diesel fuel or large propane cylinders) that can run out, then need someone to deliver more to you.  If natural gas service goes out, there are probably more serious problems to worry about...

Installing a standby generator is not something that just anyone can do.  You should enlist the help of a licensed electrical contractor, ideally one that specializes in generator installations.  Some generator manufacturers will not even warranty the work unless a "certified installer" has done the initial start-up process.  Additionally, you may need more then a basic electrical permit.  In some areas of the country, you may need a "stack permit" based on the size of the generator.  You may also need a construction permit to build a proper area to run the generator in.  The contractor you pick should know all these rules and regulations, and help guide you in the process.

Once you have the generator, here's somethings you need to do to maintain and test the generator:

• Keep records of the weekly test.  It's best to actually check the generator during it's test and make sure it works OK.  Most generators typically have some kind of diagnostic panel, either with the generator itself or as part of it's transfer switch.  Make notes of any errors at the time of the weekly test, and get them resolved ASAP.
• Have the generator serviced according to the specs provided by the manufacturer.  This typically includes an oil & filter change every 6 months, battery changes every 2-3 years, and possibly output tuning.  Remember, at it's heart a generator is basically a big engine, just like a car or large truck.  They need to be taken care of the same way you take care of a car engine.
• Do a full transfer test at least every 6 months.  This means turning off the "main" power, letting your equipment run on UPS power shortly, while the generator starts and the transfer switch switches over to generator power.  This ensures you whole power system is working properly.  If something fails, you may "go down", but at least you will be there to remedy it as fast as possible.  For example, if the generator does not fire, you can quickly turn "street power" back on.

Similarly, it's best to have as many devices with redundant power supplies as possible.  This covers you against a UPS failure at inopportune times.  With a device with RPSs, you want to avoid plugging both into the same UPS.  This is a very common mistake, which was even perpetuated by computer manufacturers like Dell, which sold "Y" power cords: one plug to two power supplies.

And while this isn't exactly the right article for this note, but I'll put it here anyway.  Lots of people, myself included, like managed / switch power strips / PDUs, that allow someone to control if an outlet is on or off to switch something on or off.  These are really handy, but I can't tell you the number of them that I've replaced one because it failed and all the outlets were stuck off.  Be careful...these can bite you in your gluteus maximus.

Remember, if you can't power your equipment, you're dead in the water and there's nothing you can do about it.  You can't count on going and buying a generator in the event you need one (since everyone else will be doing the same), nor can you count on a rental company delivering one to you when you need it.  The only way to make sure you're going to have power is to rely on yourself with your own standby power sources.

For Part 2, I'm going to talk about remote access / remote management.  Stay tuned!