A power outage at a data center owned by payment processor Total Systems Services left thousands of Canadian customers across the country unable to use their Visa cards Jan. 29.
The outage affected both the Royal Bank of Canada and TD Canada Trust. RBC issued a Tweet at 12:14 PM PT that noted problems with customer service phone lines, self-service and credit card transaction systems. TD Canada Trust also told a customer that “Visa is experiencing issues all across Canada and are working towards a resolution,” Global News reported. CIBC was also reportedly affected, according to The Toronto Star.
Total Systems Services operates a Twitter account, which it hasn’t updated since 2010, and the company is not on Facebook. Spokesman Cyle Mims, however, confirmed the outage.
“A power interruption at one of our North America data centers impacted our ability to provide payment authorizations for some of our North American clients for a period of time this [Monday] afternoon,” Mims wrote in a statement emailed the evening of Jan. 29. “At this time, systems are up and running and authorizations are being processed normally. During the outage, our operational contingency plans went into effect with the payment schemes standing in for authorizations on our behalf.”
Mims didn’t provide details about how many users were affected by the outage. Visa Inc. could not be reached for comment. Total Systems Services says that it processes payments 38 million times per day, serving 400 clients in 85 countries.
Mary Shacklett, president of market research firm Transworld Data, has written that two separate data centers can be clustered together to keep transactions at payment processors flowing.
“The goal is to parallel process data on two separate processors in two separate facilities simultaneously, so that if you need to failover from one to the other, there is no disaster recovery and backup. You simply keep going,” Shacklett wrote in a posting. “This is accomplished by clustering machines together into what appears to be a single system image that allows for data sharing and parallel computing. This is run in combination with systems management software capable of managing the end-to-end computing and data resources and performing automated failovers.”
In theory, that could limit downtime. But are such measures worth the cost? Canadian Visa customers would likely say, “Yes.”