[caption id="attachment_12710" align="aligncenter" width="500"] Hurricane Sandy devastated New York City.[/caption] On October 29, 2012, Hurricane Sandy pummeled the U.S. East Coast, hitting Manhattan particularly hard and causing mass evacuations throughout the city. Not everyone left though; two of the PEER 1 Hosting team members stayed at our Manhattan datacenter in case we lost utility power and needed to switch to our backup generators. We didn’t expect the storm to have a significant impact on the datacenter, but when my New York datacenter manager, Mike Mazzei, called shortly after the surge hit, I knew Sandy was much worse than we had anticipated. Not only had utility power gone out, but the building’s basement, where the main incoming utility switchgear and fuel tank are located, was flooding. We also only had a limited amount of fuel in the generator header tank on the 18th floor to power the backup generators on the 17th floor. Needless to say, the team was left scrambling to find a way to keep the datacenter running. After a moment of chaos and a bit of shock at the impact of Sandy, we pulled ourselves together and developed a plan to keep the Manhattan facility running. Aided by customers and a few other local PEER 1 Hosting employees who were able to get to the datacenter, Mike and his team started carrying fuel up the stairs to the 17th floor in a makeshift bucket brigade. It wasn’t a perfect solution, but they kept the bucket brigade in place for days until the building was able to install a temporary fuel hose up to the header tank that feeds the generators. When the storm cleared, our datacenter was one of the few to have maintained power. PEER 1 Hosting has datacenters in other disaster-prone areas of the country—Miami, for instance—and we’ve seen our fair share of storms. Hurricane Sandy was not our first hurricane, but we learned a lot from the unique challenges it presented us with. Not only did we learn how to best transport fuel up 16 flights of stairs, but we also learned how to prevent ever having to do it again. As we prepare for the 2013 hurricane season, there’s much the industry can take away in terms of lessons learned about how we prepare our datacenters for these large storms. It’s important to note that there are some things that datacenter managers simply can’t plan for, no matter what. For instance, it would be extremely difficult to move the main fuel tanks out of the basement and up to a higher floor. While this may prevent future disasters from affecting Manhattan datacenters as much as Sandy did, there are issues of structural support on higher floors, not to mention a higher rent rate for storing infrastructure that isn’t used very often. The goal, then, should be to prepare as much as possible to avoid being in a position where we have to react. And there’s a lot that datacenter managers can plan for to prepare for future natural disasters, including:

• Get the right equipment on-site or quickly accessible. That potentially includes roll-up generators, fuel hoses and other vital datacenter equipment, but it also means equipment for the data center teams such as flashlights and sleeping bags. In preparation of a storm, ensure there is non-perishable food and water in the facility so the crew can stay there for an extended period of time if needed.

• Have contracts in place in advance for maintenance and fuel that may be required during and after the storm. Without these contracts, you’ll be waiting hours or, worse, days until someone shows up with supplies and assistance for your datacenter (and by then, you’ll probably have shut everything down anyway).

• Staff up before a storm hits. PEER 1 Hosting wasn’t able to fly in any additional employees until the airport opened a few days after Sandy hit, so we were extremely lucky that our customers came to the Manhattan datacenter the morning following the storm to help. We would have had to do a controlled shut down of the data center if we didn’t have all of the extra hands. Consider setting up emergency response teams in the areas where you have datacenters. Those teams can be deployed at a moment’s notice to help with an outage, fire, earthquake or other natural disaster so, immediately, you have a team ready to go when they’re needed.

• If you’re a service provider like PEER 1 Hosting, make sure to also help your customers prepare for disasters because they are storing their critical infrastructure and data in your facility. Work with them to set up a disaster recovery solution to prepare for the worst case scenario. It’s also important to help your customers get through the storm without sacrificing performance, especially if you think your facility is going to go down. Post updates on your website or contact customers directly as frequently as possible to keep them updated on your facility’s status, and let them know when they should prepare for a controlled shut-down. This transparency will go miles in benefiting your relationship.

Beyond these preparations, avoid relying on everything working or going according to plan. As much as you plan and think you are prepared, it is impossible to be completely ready for a disaster in your datacenter. It’s our policy and best practice to fully test the critical infrastructure supporting data center and stock extra supplies on-site before a storm. Unfortunately these precautions weren’t enough preparation for a storm the magnitude of Hurricane Sandy. We had to adapt our hurricane procedures and how we operated on the fly to survive the storm and its aftermath. Realizing that things rarely go according to plan during a disaster is the biggest lesson to be learned from Hurricane Sandy, and one we certainly will never forget.   Ryan Murphey is VP Datacenter Operations at PEER 1 Hosting. With more than 13 years of experience in datacenter management and customer service management within the hosting and network industry, Murphey currently directs the design, budget and capacity planning for PEER 1 Hosting's 19 datacenters across the United States and Canada. Image: Anton Oparin/Shutterstock.com