Cloudflare Outage Disrupts Multiple Services Due to Phishing Block Mishap

A widespread Cloudflare outage occurred due to an error in blocking a phishing URL on the company's R2 object storage platform, causing disruptions across multiple services for nearly an hour.

The Incident: A Simple Action with Major Consequences

Cloudflare R2, a cloud-based object storage service designed for cost-effective data storage, experienced an unexpected downtime after an employee mistakenly disabled the entire R2 Gateway service instead of blocking a specific endpoint associated with a phishing complaint.

According to Cloudflare's post-mortem report, the outage happened during an abuse remediation effort when an employee attempted to take down a phishing URL. However, instead of isolating the reported URL or endpoint, they mistakenly disabled the entire R2 Gateway service, affecting critical operations.

"This was a failure of multiple system-level controls (first and foremost) and operator training," Cloudflare admitted in its official statement. The outage lasted for 59 minutes, from 08:10 to 09:09 UTC, impacting various Cloudflare services that depend on R2.

Services Affected by the Outage

• The accidental shutdown led to a 100% failure in multiple services, including:
• Stream - Video uploads and streaming delivery completely failed.
• Images - Users were unable to upload or download images.
• Cache Reserve - Caused increased origin requests due to operational failure.
• Vectorize - 75% failure in queries and a complete breakdown in insert, upsert, and delete operations.
• Log Delivery - Data loss was reported, with up to 13.6% data loss for R2-related logs and 4.5% for non-R2 delivery jobs.
• Key Transparency Auditor - 100% failure in signature publishing and read operations.

Several other services suffered partial failures, including:

• Durable Objects - Error rate increased by 0.09% due to reconnections after recovery.
• Cache Purge - HTTP 5xx errors increased by 1.8%, with a 10x latency spike.
• Workers & Pages - Deployment failures affected a small number of projects using R2 bindings.

Root Cause Analysis: Human Error and Lack of Safeguards

• Cloudflare's investigation revealed that human error was the primary cause of the outage. The absence of validation checks for high-impact administrative actions allowed the mistake to go unchecked, leading to the unintended disabling of an essential service.
• To prevent similar incidents in the future, Cloudflare has taken immediate corrective actions:
• Removed the ability to disable core systems from the abuse review interface.
• Implemented restrictions in the Admin API to prevent accidental service shutdowns.
• Strengthened internal security measures to prevent inadvertent service disruptions.

Future Preventative Measures

Cloudflare has announced further steps to ensure better control and oversight of high-impact administrative actions:

Improved Account Provisioning - Implementing more robust provisioning protocols to minimize risks.

Stricter Access Controls - Tightening security access to critical functions to prevent unauthorized actions.

Two-Party Approval Process - Introducing a dual-approval requirement for high-risk actions to add an extra layer of protection.

Cloudflare's History of Outages

This is not the first time Cloudflare has faced a major outage. In November 2024, Cloudflare suffered a significant downtime lasting 3.5 hours, which resulted in an irreversible loss of 55% of all logs. That particular incident was caused by cascading failures in the company's automatic mitigation systems after an incorrect configuration was pushed to a key logging pipeline component.

While Cloudflare remains one of the most widely used and trusted providers of internet security and performance services, these incidents highlight the importance of rigorous internal controls and better safeguards to minimize disruptions for millions of users worldwide.