When large chunks of Manhattan were in darkness a month ago after Hurricane Sandy ripped along the East Coast of America and took out electricity suppliers, New York’s data centre community took a direct hit.
At least eight New York data centres struggled with connectivity and service issues and flooding submerged and destroyed diesel pumps, preventing fuel from being pumped to back-up generators. Some large organisations, including the Huffington Post, went offline and the New York Stock Exchange shut down for a number of days.
Around 650,000 householders and businesses in NYC and Westchester County were without power. Many New York and New Jersey data centres (those that were not flooded and able to run on generators) were still running on generator power several days later, which in itself brought problems in replenishing fuel supplies as delivery lorries could not get through the floods.
Criticism has been levelled at failure to implement back-up power supplies properly so that back-up generator switchover is successful. Many data centres have generators located on higher levels than fuel storage and pumps, which means that although the generators themselves were not flooded, fuel lines were. In one case, crews of staff had to carry diesel fuel in buckets up 17 floors!
Rather than pointing the finger of blame, it’s better to look at what lessons can be learnt from this. There are some disasters that just can’t be imagined and therefore cannot be prevented, which is why data centre emergency planning is as important as having power protection UPS (uninterruptible power supplies) and emergency standby generators onsite –preferably somewhere that won’t flood but that’s not always possible in built up, coastal cities. Beyond that, for data centres, it’s a case of having back-up locations as part of your emergency response plan.
They say that disaster brings out the best in people and it’s refreshing to see that kindness does exist. Michigan company Waveform Technology allowed companies affected by the storm to install their servers on its colocation floor and keep them there for two months – free of charge!
When looking to install back-up standby power generation, it’s important to talk to your supplier, such as Riello UPS, about emergency response. The company is one of the most experienced in the business and offers more than just the hardware.