Global Outage Explained by Cloudflare CEO

Key Highlights

• Outage Details: A major outage affected numerous websites and services, including X, ChatGPT, and Spotify.
• CEO’s Apology: Matthew Prince described this outage as the worst since 2019 and apologized for the disruption.
• Cause of Outage: An issue with the system protecting against DDoS attacks led to significant failures across Cloudflare’s network.
• Restoration Time: Services were largely restored within three hours and fully restored in approximately five hours.

A Cloudflare outage took out a large swathe of the internet on Tuesday, with users unable to access numerous sites and services such as X, ChatGPT, Spotify, YouTube, and Uber. The cybersecurity company has now published a blog post detailing exactly what happened.

Cloudflare co-founder and CEO Matthew Prince apologised in the post late Tuesday, stating that this outage was the worst the company has experienced since 2019.

“[I]n the last 6+ years we’ve not had another outage that has caused the majority of core traffic to stop flowing through our network,” said Prince. “On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.”

Prince explained that the Cloudflare outage had been caused by an issue with the system it uses to protect websites from DDoS attacks.

What caused Cloudflare’s outage?

The outage was caused by a problem in Cloudflare’s Bot Management system, which protects websites from malicious bot attacks. This includes DDoS attacks that flood websites with excessive traffic.

Cloudflare’s Bot Management system is a service which protects websites against malicious bot attacks. These include DDoS attacks that flood websites with excessive traffic, content scraping attacks which gather data from websites without authorisation, and autonomous credential stuffing attacks which try to gain access to websites by using leaked login details from other sites.

This Bot Management system includes an AI model which scores traffic requests. Whenever there’s an attempt to access a website protected by Cloudflare’s Bot Management, the AI generates a score to determine if it’s likely to have been from a bot. In order to do so, the AI considers various features of the request, which are held in a “feature file.”

The feature file is where the issue occurred. This file refreshes every five minutes to keep up to date with evolving bot behaviours, and is used across Cloudflare’s entire cybersecurity network. However, the company implemented a change to the underlying query that generated the file, which caused it to duplicate information a large number of times. This made the feature file larger than typical, triggering an error in the Bot Management system.

As a result, attempting to access websites which use Cloudflare’s Bot Management system resulted in an error code. Cloudflare states that its network began experiencing significant failures about 15 minutes after the feature file generation update was implemented.

Cloudflare initially suspected the outage was a malicious attack, particularly as its status page went down despite being independent from the company’s infrastructure. However, Prince stated that this turned out to be a coincidence.

“The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,” Prince stressed. “After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file.”

When previously reached by Mashable prior to the blog post, a Cloudflare spokesperson also emphasised that “there [was] no evidence that [the outage] was the result of an attack or caused by malicious activity.”

Cloudflare’s services were largely restored within three hours, and fully restored after approximately five hours. Prince stated that the company is already planning measures to prevent similar outages in the future, including stopping error reports from being able to overwhelm its systems.