Problems in New York’s data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements.
The outages had taken several sitesoff line, most notably the Gawker network, whose hosting service, Datagram, suffered outages.
At about 7 PM on Oct. 29, local power utility ConEd began proactively shutting off power to parts of lower Manhattan, in an attempt to remove the load from the energy grid. Many data centers shifted to backup power as a result. Unfortunately, the storm’s storm surge was higher than expected, and the basements housing the generators flooded, rendering them inoperable. For the most part, these data centers then turned to alternate sources of power, typically generators that they hauled in themselves. (ConEd has provided a map of outages and the expected times when service is restored; Telx, one company affected by the outage with a facility at 60 Hudson St., said ConEd would restore power within 2 to 4 days.)
The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It’s also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services.
Below is a list of some of the data centers and services in the area, and how they’re faring:
The biggest problem for Atlantic Metro is its LGA1 facility at 325 Hudson St. The center was brought online around 3 AM local time, but suffered a generator breakdown at about 6:43 AM. Engineers are investigating. AM’s LGA3 site is up and running at 111 8th Ave.
“LGA4 IDC Update: Site is suffering from a damaged ATS. Crews will be called in first thing Wednesday morning to begin repair efforts,” the company tweeted.
“Authorities have given the OK to allow access into our NJ3 facility at 5851 Westside Ave, North Bergen, NJ” the company said at about 3:30 PM on Oct. 30. No further updates have been given.
No issues. “UPDATE: NYC data center remains in a stable condition on emergency generator power, fuel tanks have been topped off,” the company tweeted at 4 PM Tuesday.
As of 8 AM Eastern, Datagram’s facilities had nearly pumped all of the water out of the 33 Whitehall facility. As soon as ConEd gives the all-clear, the Datagram generator can be restarted. Otherwise, the company said it had rented an alternate generator that could be placed outside. “Roll up generators are not allowed in the area until all the water is removed,” the company said.
Equinix appears to be back on track. The company reported that “Four of our sites have returned to utility power (NY1, NY7, DC7, PH2) while another six remain on generator power,” it said in a blog post. “We have ample fuel onsite and fuel vendors available to deliver more as needed to the remaining sites. All reported water leaks have been contained. There is no impact to customers at this time.”
Status unknown. The company has a facility at one of the major connectivity hubs at 67 Broad St., but hasn’t issued any updates.
Internap operates two facilities, one at 75 Broad St., and one at 111 8th Ave. The Broad St. facility was flooded with up to three feet of water and shut down, and the company said late on Oct. 30 that it was “working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation,” it said. “It is unclear how long it will take ConEd to restore utility power to the site, but we are preparing for the possibility of remaining on generator power for many days.
“Our 111 8th Ave. data center continues to run on generator power successfully,” the company added. “We have enough fuel to last several days and expect fuel delivery to the site will be possible prior to depleting on-site reserves. We continue to see many network issues throughout the region, with many vendors losing either power or facilities due to flooding or the like.”
Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs.
“Word from our troops on the ground suggests that we should not see an outage due to lack of fuel before Con Ed can restore power to the building. This is NOT guaranteed, but things are looking good,” Fog Creek said in a status update. “The generator tank has about 2 hrs of fuel in it on the 17th floor. The bucket brigade has 8 barrels of fuel to carry up, or enough for ESTIMATED 15 hours of fuel. Since the diesel truck is tentatively staying available for the duration, our fuel concerns may be addressed. Provided a continuing supply of bucket-brigade labor, no outage is expected before power is restored.”
But another customer, Squarespace, said that pumping of the basement water is going “very slowly.”
All sites are up and running, although some are on generator power. The company’s NYC1, NYC2, and NJR1 facilities are all running on diesel-fuel-powered generators, but all three sites should have ample fuel, with additional fuel deliveries scheduled for midday, the company said. All other facilities are operating nortmally. However, the company advised customers to power off non-essential equipment if possible.
No outages. All Telehouse New York City facilities proactively engaged backup diesel power (off-grid) in anticipation of lower Manhattan’s utility power shutdown,” the company said. “Since that time, these facilities have remained operational without any critical system (power, cooling) outages. These facilities will remain fully staffed and off-grid (independent power generation mode), with periodic refueling until utility power is restored and stable.”
“Verizon restoration efforts are well underway in the Mid-Atlantic and Northeast regions, as Sandy moves further north,” Verizon said in an update at 8 AM on Wednesday. “The company’s networks are functioning relatively well, despite intensive efforts to clean up after massive flooding and the continued lack of commercial power in many locations up and down the Eastern seaboard.”
“We continue to have significant issues in Lower Manhattan. Con Edison reports that it may take a week to restore power to parts of Manhattan, and removing contaminated floodwaters requires monitoring from environmental agencies, both contributing to restoration timeframes.”