[caption id="attachment_5656" align="aligncenter" width="618"] ConEd outage update.[/caption] The local utility serving most of the New York City area, Con Edison, reported that it should begin supplying utility power to midtown and lower Manhattan by Saturday evening, returning the island’s data centers and citizens to some semblance of normalcy. For some reason, ConEd didn’t add that bit to its Nov. 2 update, instead sharing the information via Twitter. The exceptions will be buildings in Lower Manhattan that suffered damage as a result of the hurricane (possibly as many as 100) and need repairs. Virtually all ConEd customers should have power by Nov. 11, the company said, due to a high number of downed trees in Westchester County and other, more rural areas. In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power. That should come to an end, for the most part, when utility power is restored. In a possibly worrying note, Verizon warned late on Nov. 1 that its services to business customers could be impacted due to lack of fuel. Below is the Nov. 2 update from many of the data-center providers and some of the communications providers in the area, some of which remain unchanged from previous days.

AtlanticMetro:

Atlantic Metro reported all of its facilities are now up and running, although the two Manhattan areas, LGA1 at 325 Hudson St., and LGA4 at 121 Varick St., remain on generator power. LGA1 has 5,000 gallons of fuel on hand, with 3,500 gallons committed delivery for Saturday night. LGA4 is somewhat more precarious, with a rollup generator on hand, with a 250-gallon belly tank and a 300-gallon on-site reserve tank. The NJ3 location is on utility power and operating normally, the company said.

AT&T:

The company didn’t provide updates on its data centers or business services, reporting on the progress of its wireless services. AT&T said it would donate $250,000 to the Red Cross for relief efforts. The company also plans on providing several mobile charging stations in affected New York City boroughs from 1 to 5 PM EST on Friday for residents to charge phones. The company has also waived late-payment and non-payment penalties, and signed a roaming agreement with T-Mobile.

CoreSite:

No issues and no updates.

Datagram:

Datagram still does not have a generator on site to provide emergency power. At 6 AM EST on Nov. 2, the company issued the following update:

“Damage reports are still coming in. Fuel in basement, must be removed/cleaned. Water in fuel tank, needs replacement. Virtually all switch equipment has been replaced. Electricians are working around the clock. Generator is near by but city is not allowing it closer until area is deemed safe. We are fighting to get it placed for a few hours. We have permits in hand. Our hands are being tied by the City, ConEd and building management.”

At 8:42 AM EST, the company added:

“Generator is on 26th Street and heading downtown now. All of our efforts are focused on getting this placed and hooked up. We understand you want more updates and answers but this what we have. Generator is very very close to being parked. All preparations have been made for electrical, we are ready to hook up. Pump restoration is beginning this morning.”

Customers were still using alternative sites to deliver updates.

Equinix:

Equinix appears to be stable but has suffered some generator problems that have affected service. “All of our New York and New Jersey sites are accessible to customers,” CoreSite said at 3:30 PM EST on Nov. 1, its latest update. “Three of those sites (NY1, NY7, and NY10) are back on utility power and are operating normally. The other five sites (NY2, NY4, NY5, NY8, and NY9) remain on generator power. All of these sites have ample fuel onsite, and we have successfully received fuel deliveries over the past two days with regular scheduled deliveries to each site until utility power is restored. “This morning our NY8 site experienced a failed generator that impacted service to several customers,” CoreSite added. “The generator was restarted and service has been restored. Over the past two days our NY9 site, which is housed in a multi-tenant building with shared generator infrastructure and fuel service, has experienced three outages due to generator failures that impacted service to several customers. Repairs have been made and service was restored. The site remains on generator power.” Sites in Philadelphia and Washington D.C. are operating normally on utility power.

Ezzi:

Ezzi operates three facilities: at 882 Third Ave. in Brooklyn, and at 75 Broad St. and 25 Broadway. The company has not published any updates on its Web site or Twitter feed.

FiberMedia:

Status unknown. The company has a facility at one of the major connectivity hubs at 67 Broad St., but hasn’t issued any updates. Company representatives did not reply to an email requesting a status update at press time, but the company has provided a support page, with a dedicated Sandy support number, for customers.

Internap:

Internap operates two facilities, one at 75 Broad St., and one at 111 8th Ave. At 111 8th (NYM008/LGA9), the facility continues to operate on stable generator power, Internap reported. The fuel supply at this site is provided by the building landlord, and is shared with all tenants in the building. At this time, the fuel supply is estimated to last approximately three days. The site expects to receive 5 – 6 fuel deliveries each day for a total of approximately 25,000 gallons. At 75 Broad (NYMEXT1/LGA11), the facility also continues to operate on stable generator power. The site currently has roughly 12,000 gallons of fuel, which is estimated to support it for approximately 7+ days. Fuel suppliers are on standby for additional deliveries as needed until utility power is restored. The LGA6 / LGA8 facility is also operational on stable generator power. “While the fuel supply is shared at this facility, we would like to confirm that we are NOT experiencing any issues with Internap generators or pumps onsite at this time,” the company said in an update.

Level3 Communications:

“To this point, we have experienced no major service disruption. There have been a few isolated incidents affecting a small number of customers, and we are resolving these as quickly as possible,” the company said late on Oct. 30. The company has issued no further updates.

Peer1:

While a bucket brigade that carried diesel fuel up 18 flights of stairs to the generators that kept Peer 1 operations running, the company has so far refused to provide updates, preferring to let its customers provide on-the-ground communications. In a status update provided at 9 PM EST local time Nov. 1, Fog Creek noted that the situation has stabilized, and that the company’s services were down for three hours on Tuesday, a self-imposed shutdown to fight data corruption. “Barring a completely new problem, the only further post here will be to let you know that our data center is back on the grid,” the company said. Squarespace has had its own share of problems, including ferrying fuel upstairs. Now, the company says that it has connected a fuel pump to the roof, which will allow indefinite fueling. However, the fuel pump filter will need to be cleaned at some point, which will result in about an hour’s downtime unless Peer1 can provide a street-level generator, which apparently hasn’t happened.

Telx:

Telx said its status of its New York data centers were virtually unchanged on Nov. 2. All sites are up and running, although some are on generator power. Supplier Taconic Management has received an estimated 12,000 gallons of fuel, with several deliveries to the NYC2 site are expected throughout the day estimating a total of 20,000 gallons to be delivered. Taconic has reported that the fuel supply is currently at approximately 40,000 gallons, Telx said. The NJR1 site is also on generators, with “adequate” fuel levels, Telx reported. As before, the company advised customers to power down where they could, in part because temperatures at the 111 8th Ave. (NYC2) facility soared past 100 degrees as generator problems forced a cooling system shutdown.

Telehouse:

No outages and no updates for Nov. 2. Telehouse is now offering colocation space, telecom circuit cabling and IT Support services on a temporary basis (or limited basis) at all three of the company’s New York City Data Center facilities: 25 Broadway, Chelsea at 85 10th Ave., and the Teleport in Staten Island. Quarter, half and full cabinets are available on temporary one month or more basis over the next 60 days, in addition to long term commitments.

Verizon:

Verizon’s infrastructure services appear to be operational, but the company is warning that business customers could be affected due to lack of fuel. “While our buildings and facilities depend on commercial power and we are working closely with the local power companies to prioritize our needs, it will be difficult to have every system up to pre-Sandy performance until commercial power is restored,” the company said at 4 PM EST on Nov. 1. “Verizon builds redundancy into our facilities so that if things go wrong we can continue to service our clients,” the statement continued. “Back-up power at our facilities is key to redundancy planning.  Generators are a viable temporary solution, but they are not designed for lengthy commercial power outages caused by major catastrophes. In some locations, we have not received adequate fuel due to power generators.  When generators run out of fuel, our facilities switch to battery power, which provides limited relief. We are aggressively working to obtain commercial power.” However, Verizon said that it “brought back up a key facility in lower Manhattan, which will help service our business clients as they return to work in the area.”   Image: ConEd