Crashes suck.[/caption] CloudFlare essentially "dropped off the Internet" for about an hour on March 3, as all its edge routers (used by its 23 worldwide data centers) failed. Why? Bad code. Part of the problem was that CloudFlare received poor information. That morning, as one of the company's customers was hit by a DDoS attack, the company's internal tool for detecting and profiling attacks returned some odd data: packets involved in the attack were between 99,971 and 99,985 bytes long—and that's strange, as CloudFlare noted, because the largest packets sent across the Internet are typically in the 1,500-byte range and average between 500 to 600 bytes. CloudFlare's edge routers are manufactured by Juniper and support a tool, FlowSpec, which allows a customer to push new router rules very quickly. When CloudFlare told the routers to disregard packets of between 99,971 to 99,985 packets long, they crashed, consuming every bit of RAM. Worse still, they were either unable to reboot or crashed again when they restarted. "We removed the rule and then called the network operations teams in the data centers where our routers were unresponsive to ask them to physically access the routers and perform a hard reboot," CloudFlare wrote, according to a Google cache of the page. (CloudFlare's blog was inaccessible Monday morning.) "We have already reached out to Juniper to see if this is a known bug or something unique to our setup and the kind of traffic we were seeing at the time," CloudFlare added. "We will be doing more extensive testing of Flowspec provisioned filters and evaluating whether there are ways we can isolate the application of the rules to only those data centers that need to be updated, rather than applying the rules network wide.” The company will also “proactively issue service credits to accounts covered by SLAs. Any amount of downtime is completely unacceptable to us and the whole CloudFlare team is sorry we let our customers down this morning." CloudFlare noted that its own error mirrored what happened in Syria earlier this year. Only in this case, the outage was accidental, not malicious—but it was an outage just the same.
CloudFlare, Microsoft Azure Explain Outages
[caption id="attachment_8171" align="aligncenter" width="500"] Crashes suck.[/caption] CloudFlare essentially "dropped off the Internet" for about an hour on March 3, as all its edge routers (used by its 23 worldwide data centers) failed. Why? Bad code. Part of the problem was that CloudFlare received poor information. That morning, as one of the company's customers was hit by a DDoS attack, the company's internal tool for detecting and profiling attacks returned some odd data: packets involved in the attack were between 99,971 and 99,985 bytes long—and that's strange, as CloudFlare noted, because the largest packets sent across the Internet are typically in the 1,500-byte range and average between 500 to 600 bytes. CloudFlare's edge routers are manufactured by Juniper and support a tool, FlowSpec, which allows a customer to push new router rules very quickly. When CloudFlare told the routers to disregard packets of between 99,971 to 99,985 packets long, they crashed, consuming every bit of RAM. Worse still, they were either unable to reboot or crashed again when they restarted. "We removed the rule and then called the network operations teams in the data centers where our routers were unresponsive to ask them to physically access the routers and perform a hard reboot," CloudFlare wrote, according to a Google cache of the page. (CloudFlare's blog was inaccessible Monday morning.) "We have already reached out to Juniper to see if this is a known bug or something unique to our setup and the kind of traffic we were seeing at the time," CloudFlare added. "We will be doing more extensive testing of Flowspec provisioned filters and evaluating whether there are ways we can isolate the application of the rules to only those data centers that need to be updated, rather than applying the rules network wide.” The company will also “proactively issue service credits to accounts covered by SLAs. Any amount of downtime is completely unacceptable to us and the whole CloudFlare team is sorry we let our customers down this morning." CloudFlare noted that its own error mirrored what happened in Syria earlier this year. Only in this case, the outage was accidental, not malicious—but it was an outage just the same.
Crashes suck.[/caption] CloudFlare essentially "dropped off the Internet" for about an hour on March 3, as all its edge routers (used by its 23 worldwide data centers) failed. Why? Bad code. Part of the problem was that CloudFlare received poor information. That morning, as one of the company's customers was hit by a DDoS attack, the company's internal tool for detecting and profiling attacks returned some odd data: packets involved in the attack were between 99,971 and 99,985 bytes long—and that's strange, as CloudFlare noted, because the largest packets sent across the Internet are typically in the 1,500-byte range and average between 500 to 600 bytes. CloudFlare's edge routers are manufactured by Juniper and support a tool, FlowSpec, which allows a customer to push new router rules very quickly. When CloudFlare told the routers to disregard packets of between 99,971 to 99,985 packets long, they crashed, consuming every bit of RAM. Worse still, they were either unable to reboot or crashed again when they restarted. "We removed the rule and then called the network operations teams in the data centers where our routers were unresponsive to ask them to physically access the routers and perform a hard reboot," CloudFlare wrote, according to a Google cache of the page. (CloudFlare's blog was inaccessible Monday morning.) "We have already reached out to Juniper to see if this is a known bug or something unique to our setup and the kind of traffic we were seeing at the time," CloudFlare added. "We will be doing more extensive testing of Flowspec provisioned filters and evaluating whether there are ways we can isolate the application of the rules to only those data centers that need to be updated, rather than applying the rules network wide.” The company will also “proactively issue service credits to accounts covered by SLAs. Any amount of downtime is completely unacceptable to us and the whole CloudFlare team is sorry we let our customers down this morning." CloudFlare noted that its own error mirrored what happened in Syria earlier this year. Only in this case, the outage was accidental, not malicious—but it was an outage just the same.
