At the end of October, Hurricane Sandy struck the eastern seaboard of the United States, leaving massive amounts of property damage in its wake. Data center operators in Sandy’s path were forced to take extreme measures to keep their systems up and running. While flooding and winds knocked some of them out of commission, others managed to keep their infrastructure online until the crisis passed.
In our previous interview, we spoke with CoreSite, a Manhattan-based data center that endured even as much of New York City went without power. For this installment, Slashdot Datacenter sat down with executives from IPR, which operates two data centers—in Wilmington, Delaware and Reading, Pennsylvania—close to Sandy’s track as it made landfall over New Jersey and pushed northwest.
This interview has been edited slightly for content.
Just for the record, what’s your name and title?
Richard J. Cowell, and I run the business development effort for IPR.
How many data centers do you operate?
We have two. They’re mirrored. One in Reading (Pa.), 60 miles away from Wilmington.
And Hurricane Sandy affect them both?
Both were not affected, although we were in the eye of the hurricane.
So you suffered no downtime as a result?
Let’s define “data center,” first of all. There are a lot of different data centers, and they’re certified by tier. Both of our data centers are Tier 3+. The highest is Tier 4. Our data centers are “2n,” which means two of everything. So redundancy is the key to any disaster. We have two power grids we sit on top of, so if the power goes we flip over immediately to [backup]. We have two generators, we have multiple fuel tanks that are full, and we’re built to be 24x7x365. Since we’re a professional disaster recovery firm, our people test, simulate, and practice all year round for an event like this. Not like our data centers, or other folks caught by surprise.
This disaster—an absolute tragedy, and I have friends up in Manhattan—is an absolute validation of our business model, investment and processes. That’s really what it is for us.
So you had plans in place?
Of course. We’re audited by the Federal Reserve; the TSA does background checks on our folks—we’re built for this sort of thing.
So when Hurricane Sandy comes on your radar, so to speak, what actions did you take?
The biggest action is pre-emptive. Notifications of our customers, notifications of our prospects, Web page alerts, from our Zone Service Centers. We just put up the actual alert that went out to our customers. It’s a weather map, it’s reporting the impact probability and statistics, all the inclement weather alerts, buckle down the hatches, IPR’s here, we’re here by your side.
But there’s perpetual communication throughout the entire disaster to our clients. Day in and day out, and there’s a summation at the end. What we end up getting is a lot of folks who aren’t our clients, by referral, and we’ve had several of those who said, “We’re out in Battery Park, New York, what can you do to help? Because our commerce revenue is stopped.” And we put in a VPN in a day, and we worked all the last three weekends on these types of left-field requests… We work out the contract specifics later. We had about six of those.
And were there any eventualities that you didn’t plan for? Anything not covered?
We were very pleased. The staff was there. People worked a little extra. We weren’t working 9 to 5, so people got bleary-eyed, but everybody was on the decks, full coverage. Not a blip.
Were you ever on backup generators?
We were never on backup generators. Not once. We’re proud of it.
And was that due to the local utility?
That’s part of our investment. We potentially have two power grids. We actually have three. The third we would never go to, I don’t think. And it’s like a circuit breaker. If one goes down, the other flies up immediately. And our data center in Wilmington, three and a half years ago, it only happened one time, and it happened for less than three and a half minutes. We journal all those events.
And it didn’t happen during Sandy?
No. Not at all. Our fuel tanks were all full, our generators were all tested—because we do that anyway, we test our generators. We were prepared if it did, but it never did. We never went to generator backup.
Did you allow customers access to the building during Sandy?
And they took advantage of it?
We have people who own the seats, where their whole livelihood depends on those cubes. So they call in those seats and bring their folks in. And then we have people, where we have first come, first served. So they might pay $30 bucks per month, per seat, but they’re sharing it with five different companies, who are proximity-wise far away from one another. So let’s say we wouldn’t put Atlantic City with Jersey City in the same seat; someone from Atlantic City sharing a seat with someone from Pittsburgh… We didn’t have any seat that was unavailable, one—and two, we did have people in our data center taking advantage of our protective environment.
Some hotels in New York ran out of water and food—I don’t know if local hotels did, too. Did you have stocks of both on hand?
We had everything on hand in the facility—showers, everything. Now it’s not a hotel, and we did those arrangements too, if they needed to come.
Going forward, would you make any changes to your contingency plans, or was what you had in place satisfactory?
Well, you know, it’s still fresh for us, even though it’s still a few weeks after. I was talking to a company today that was up in Manhattan, an investment firm, they had said, “Let’s sign a contract with you, let’s take our time.” They called last night, 11 o’clock last night, and they said, “Oh my God, the system went back down. How fast can you get me up?” So our guys were scrambling today, getting them servers, all the Internet, all the emails for all their customers, and they’re a pretty big firm. We’re getting them up by Saturday. So it’s fresh for us, and it will be for a while.
We haven’t stopped and done an assessment yet; so far our report card is pretty good.
So from talking with customers back in New York, things aren’t back to normal.
No, they’re not. They’re our customers. It’s really rough, because they don’t know what to do. Some of them, their computer systems are fine, but the power is in the bottom of the building. Let’s say they’re on the 30th floor, but the power’s in the bottom of the building so they have absolutely no power.
And I should have asked about this earlier—your communications infrastructure was never down?
None. Zero. But we have redundancy there, too. Everything has to be redundant to get [an] audit qualification annually. To get [what’s] called a 2N. Not only your power but everything: cooling has to be redundant, generators have to be redundant, you have to have redundant communication pathways, everything.
Are there any other issues that I haven’t asked about that you think are important to mention?
We never plan for this. I didn’t. And when it comes, it can be devastating. I was watching the news, and some of the [Jersey] Shore points. I have friends out there. Horrible. And the other thing is: it’s good to leave it to a professional, because we do it every day.
This isn’t a sales pitch, but if you do it yourself, you only have yourself to yell at, to complain to. And if you have someone [who’s] SSAE16 certified, with audits to see that we are simulating, we are testing, before they write-off that their clients are safe. That might not be a bad thing.
The other thing is, we’re talking about a hurricane. What scares the bejesus out of me is an earthquake. You don’t know it’s coming.
Image: Anton Oparin/Shutterstock.com