HomeIncident library
Incident library

INCIDENT REPLAY · 2026-02-20

Cloudflare BYOIP prefixes withdrawn by buggy cleanup task

A scheduled sub-task written to automate removal of BYOIP prefixes queried an API without a required value, got every prefix back, and treated them all as queued for deletion. About 1,100 customer prefixes (roughly 25% of BYOIP) were withdrawn from BGP, making those customers' services unreachable; full restoration took about 6 hours.

Fits no familyBGP withdrawal / announcementCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Cloudflare

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • A scheduled sub-task written to automate removal of BYOIP prefixes queried an API without a required value, got every prefix back, and treated them all as queued for deletion. About 1,100 customer prefixes (roughly 25% of BYOIP) were withdrawn from BGP, making those customers' services unreachable; full restoration took about 6 hours.
  • Who: Cloudflare

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • This incident rests on one or two write-ups. The cause was in an address-management system; what reached the routers was customers' prefixes being withdrawn, which is what is replayed (12 stand-in prefixes at two sites).
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
fail, for the cause: every withdrawn prefix is named per site.
Verdict now
unchanged.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the change as pushed, from the operator's own routers

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • CF-EDGE-AMS would stop announcing
  • CF-EDGE-IAD would stop announcing

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The root cause is an API/automation bug upstream of any device config, but its output, the withdrawal of about 1,100 announced prefixes, is exactly what a gate on the rendered change would refuse if the automation's pushes pass through one.

Imported from docs/replays/cloudflare-byoip-2026/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.