Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?
In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it’s become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting.
For now, power is slowly coming back to Manhattan and its wealth of data centers.
In total, 676,000 customers in the New York area served by Con Edison remain without power, including 227,000 in Manhattan. Fortunately, the utility was able to turn on the power for one network in lower Manhattan, bordered by Vesey Street on the north; West Street; Broadway and State Street on the east; and the Battery’s southern tip. The utility noted that, although it has restored power to the networks, some buildings may still be without electricity due to basement flooding or damage to local equipment. Somewhat oddly, a comparison of ConEd’s outage map for the lower Manhattan areas between Oct. 31 and Nov. 1 shows more outages today than yesterday—possibly as a result of better or more granular reporting.
Here’s the latest on providers around the New York area.
Atlantic Metro has secured its LGA1 facility at 325 Hudson St., reporting that it is 100 percent restored. As of roughly 5 AM EST, it had 2,350 gallons of fuel powering it. The company said it is working to restore Ethernet over Copper and Ethernet over Fiber services affected in the area. Overnight, the company dispatched teams to 111 8th Ave and 76 9th Ave to solve affected customer circuits.
The problem now is LGA4 at 121 Varick St., which suffered a generator failure Oct. 31. That facility is still not online, and Atlantic Metro is bringing in a rollup generator to solve the problem, which should be on site by 2 PM EST. “Once the generator arrives on site there will still be several hours of electrical work that needs to be completed to tie the unit into the bus and inspect the UPS units and properly power up the site,” Atlantic Metro said. “Our current operational timeline has us projecting site restoration to emergency power by midnight.”
No further updates were announced, including the NJ3 facility. “Authorities have given the OK to allow access into our NJ3 facility at 5851 Westside Ave, North Bergen, NJ,” the company said at about 3:30 PM on Oct. 30. No further updates have been given.
The company didn’t provide updates on its data centers or business services. “The vast majority of our cell sites in the Northeast are online and working,” it reported. “We are making progress in areas that were especially hard-hit, including New York City and New Jersey, where flooding, power loss, transportation and debris all pose challenges. We are working around the clock, including conducting ongoing damage assessment, rapid deployment of generators and equipment, and movement of key personnel from around the region and country, such as engineers and technicians, in order to restore service as quickly as possible.”
The company has waived late-payment and non-payment penalties, and signed a roaming agreement with T-Mobile.
No issues. Rob Rockwood, the senior vice president & general manager of Coresite’s Eastern Region, spoke with Bloomberg TV about how the company managed the disaster: its facility at 32 Avenue of the Americas used batteries to transition between utility and generator power. “It was a seamless transition,” Rockwood told Bloomberg.
The company then began estimating how quickly diesel fuel was being consumed, and worked out the logistics to resupply its generators.
Support teams are working with customers to migrate data or servers to other locations, the company said Oct. 31.
Via Twitter, the company confirmed fiber connectivity out of its 33 Whitehall facility; it Tweeted at 8 PM EST on Oct. 31 that “less than 2 feet of water” needed to be pumped out of the basement. “A 2 megawatt mobile diesel generator is confirmed to arrive tomorrow (Nov. 1) before 5 PM,” the company added. “There is a chance that power will be restored before 5PM.”
“It is our generator for at least one week or longer if needed,” Datagram added via its Website. “2MW is at least twice the power we need. Fuel tank will last two days. We have nightly fuel deliveries scheduled. We will keep generator on-site until we have ConEd utility power fully restored. We have offered to power the building elevators and emergency systems as well for everyone’s general safety. We do not know how long it will take to park and connect but we will be providing live updates and pictures as soon as it arrives.”
Customers such as Gawker are still using alternative sites to deliver updates.
Equinix appears to be stable, at about the same status as on Oct. 31.
“Three of our sites in New York and New Jersey (NY1, NY7 and NY10) are back on utility power,” the company said at 4:45 EST Oct. 31, its latest update. “The other five sites in the region remain on generator power. All of these sites have ample fuel onsite, and we have successfully received fuel deliveries today. We are scheduling regular deliveries to each site until utility power is restored. Early this morning, our NY9 site experienced a failed generator that impacted service to several customers. Repairs were made and service was restored within one hour and 15 minutes.”
Sites in Philadelphia and Washington D.C. are operating normally, on utility power. Via Twitter, the company apologized for its lack of public updates, but said customers have access to “an extensive communications network”.
Ezzi operates three facilities: at 882 Third Ave. in Brooklyn, and at 75 Broad St. and 25 Broadway. The company has not published any updates on its Web site or Twitter feed. When Slashdot called the support line for an update, we were told that one data center remained open and one remained down. The representative then hung up.
Status unknown. The company has a facility at one of the major connectivity hubs at 67 Broad St., but hasn’t issued any updates. As of noon EST Nov. 1, a company representative said that the company was holding a meeting with its customers, and wouldn’t be able to comment until later in the day. Unfortunately, the company is still promoting its “Market Storm” bundle on its home page.
Internap operates two facilities, one at 75 Broad St., and one at 111 8th Ave. The Broad St. facility was flooded with up to three feet of water and shut down, and the company said late on Oct. 30 that it was “working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation.”
On Nov. 1, things seem stable. The NYM008/LGA9 at 111 8th St. “continues to operate on stable generator power,” Internap said. “All UPS modules in phases 1, 2, & 3 are online and providing conditioned generator power to the critical infrastructure.” The site has enough fuel for two days, with continued fuel deliveries expected as needed until it returns to commercial power.
The NYMEXT1/LGA11 site at 75 Broad continues to operate on stable generator power, Internap said. UPS units are supplying conditioned generator power to the critical infrastructure. The building continues to pump water out of the sub-basement floors and access is limited for non-tenants. “There are currently two delivery trucks onsite to replenish the fuel supply and we will continue to bring in deliveries as needed until utility power is restored.”
As of 8 PM EST Oct. 31, Atlantic Metro reported that Internap suffered multiple power failures at the 76 9th Ave facility and that it was investigating some “fried network gear.” Internap reported that it was working through some “individual customer issues.”
Utility power should be restored in three to four days, according to ConEd.
“To this point, we have experienced no major service disruption. There have been a few isolated incidents affecting a small number of customers, and we are resolving these as quickly as possible,” the company said late Oct. 30. The company has issued no further updates.
Peer1’s operations at 75 Broad became one of the Sandy stories that made for good press: a bucket brigade that carried diesel fuel up 18 flights of stairs to the generators that kept operations running. The hosting company enlisted its own customers to help out.
But there’s bad news. Manpower appears to be running short. Worse still, the building is in shutdown mode, customer Squarespace reported at 11:16 AM EST. “We may not be able to deliver more fuel, as the building is shutdown mode. Last estimate is that we have about 3 hours left. Things change constantly, and we will keep you updated.”
Customer Fog Creek Software, meanwhile, transitioned its Trello collaboration service over to Amazon’s AWS as a preventative measure. The company had reported that the bucket brigade was running out of steam: “Our services remain up and running, but our additional helping hands to refuel the generator are being challenged,” it reported. “The good news is that we still have some Peer1, Squarespace, and Fog Creek employees at the data center to refuel as needed. All employees are on travel-standby to assist in refueling efforts.”
Earlier, at 9:03 AM EST, Squarespace also provided some fascinating on-the-ground color: “This building has 2 basements and both are flooded. As of 7AM EST today, the first basement has been completely pumped out. A broken water main was discovered which was ADDING water to the basement as the pumps ran yesterday. By using 3x the number of pumps overnight, they were able to outpace the water main, clear the basement and turn off the running water main.”
All sites are up and running, although some are on generator power. Basically, the status is unchanged from Oct. 31, with NYC1 at 60 Hudson St. and NYC2 at 111 8th Ave. running on generators. NYC1 has fuel for one day’s worth of service, with a refueling truck scheduled for Nov. 1. NYC2 has two days of fuel on hand, restocked at 4:30 PM EST on Oct. 31. The NJR1 site is also on generators, with “adequate” fuel levels, Telx reported. As before, the company advised customers to power down where they could.
No outages. Telehouse is now offering colocation space, telecom circuit cabling and IT Support services on a temporary basis (or limited basis) at all 3 of the company’s New York City Data Center facilities: 25 Broadway, Chelsea at 85 10th Ave. and the Teleport in Staten Island. Quarter, half and full cabinets are available on temporary one month or more basis over the next 60 days, in addition to long term commitments.
Verizon’s infrastructure services appear to be operational. “Because critical communications equipment (such as voice switches, data equipment and routers) are located on high floors in our buildings, they were not damaged by floodwater,” the company said the evening of Oct. 31. “But the water did damage some of backup power equipment such as switch gear, generators and fuel pumps, causing some repair delays.”
On Nov. 1, the company said little about its wired infrastructure services, choosing to focus on how it was erecting temporary wireless towers for customers, and rescue and recovery efforts.