HomeIncident library
Incident library

INCIDENT REPLAY · 2025-07-14

Cloudflare 1.1.1.1 global outage: resolver prefixes attached to a non-production service topology

On June 6 the 1.1.1.1 resolver prefixes were mistakenly included in the service topology of a new, not-yet-live Data Localization Suite service. On July 14 a change to that topology triggered a global refresh, and the resolver prefixes were withdrawn from Cloudflare's routers worldwide. 1.1.1.1 was unreachable for most users for 62 minutes.

Fits no familyBGP withdrawal / announcementCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Cloudflare

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • On June 6 the 1.1.1.1 resolver prefixes were mistakenly included in the service topology of a new, not-yet-live Data Localization Suite service. On July 14 a change to that topology triggered a global refresh, and the resolver prefixes were withdrawn from Cloudflare's routers worldwide. 1.1.1.1 was unreachable for most users for 62 minutes.
  • Who: Cloudflare

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The fault was in Cloudflare's service-topology system, which is not a router configuration. What reached the routers was the withdrawal of the resolver's prefixes everywhere, and that is what is replayed.
  • Three sites, three prefixes (stand-ins). The intended change retires one test prefix at one site.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
not runnable: --allow-withdrawal did not exist. Without it the intended change could only be waived with the blanket flag, which waives the outage as well.
Verdict now
pass for the intended change with its withdrawal named, fail for the push: each site would stop announcing the two production prefixes.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the change as intended, each withdrawal named for its router

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-withdrawal 198.18.100.0/24@CF-EDGE-AMS returned pass.

Pinned in tests/gauntlet/replays-individual.yaml.

the change as pushed, from the operator's own routers

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-withdrawal 198.18.100.0/24@CF-EDGE-AMS returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • CF-EDGE-AMS would stop announcing
  • CF-EDGE-IAD would stop announcing
  • CF-EDGE-SIN would stop announcing

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The error sat in Cloudflare's internal service-topology system rather than router configs, but the rendered effect (global withdrawal of 1.1.1.0/24 and siblings) is a before/after announcement diff a gate would refuse.

Imported from docs/replays/cloudflare-1111-2025/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.