HomeIncident library
Incident library

INCIDENT REPLAY · 2013-03-03

Cloudflare 2013: Flowspec rule crashes every edge router

While mitigating a DDoS attack, Cloudflare pushed a Flowspec rule matching an impossible packet length range; Juniper edge routers consumed all memory and crashed network-wide, taking Cloudflare offline for about an hour.

FlowspecSoftware bug / upgradeCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Cloudflare; Juniper routers

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • While mitigating a DDoS attack, Cloudflare pushed a Flowspec rule matching an impossible packet length range; Juniper edge routers consumed all memory and crashed network-wide, taking Cloudflare offline for about an hour.
  • Who: Cloudflare; Juniper routers
  • Why it is partly in remit, as the catalog records it: The crash is a vendor bug no model predicts, but the rule itself matches an impossible packet length (above any MTU and the 16-bit IP length field), which a semantic lint of the candidate rule can refuse.

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The routers: two Cloudflare edges in Junos syntax (Cloudflare ran Juniper at its edge), iBGP between loopbacks over OSPF with the flow family already enabled, one of them with an eBGP uplink to a transit router in IOS syntax. AS 13335 is Cloudflare's; every address is from the documentation ranges.
  • The rule. The catalog records only "an impossible packet length range". We write it as the post-mortem is commonly recalled, a Junos flow route matching a DNS server's /32, port 53 and packet lengths 99,971 and 99,985, with a discard action; we have not re-read those values on the page, and the address is ours. Any length above 65,535 gives the same verdict. The rule is configured on one edge; in the incident it was pushed to every edge, which is what BGP does with it.
  • The control replaces the lengths with 512-1500 and names UDP. What the attack packets actually measured is not needed and not claimed.
  • What crashed the routers was a vendor bug (memory exhausted while installing the rule). No model predicts a crash, and this replay does not claim one. What it shows is that the rule can match nothing, which is never what an operator means.
  • A reconstruction, not Cloudflare's configuration. Nothing here says the outage would have been prevented.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

the rule as pushed: packet lengths no packet can have

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • Flowspec rule 198.51.100.53/32-DNS-DROP on CF-EDGE-1 can match nothing
  • packet-length 99971 99985 (an IPv4 packet is at least 20 and at most 65,535 bytes
  • Cloudflare's edge routers on 3 March 2013

Text no reason may contain

  • BGP session
  • unparsed

Pinned in tests/gauntlet/replays-flowspec.yaml.

the same push with --allow-flow-changes: still not certified

now: unknown

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned unknown.

Blockers pinned (some blocker contains each line)

  • flowspec_effect_unmodeled
  • 198.51.100.53/32-DNS-DROP
  • packet-length 99971 99985

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

  • Flowspec rule 198.51.100.53/32-DNS-DROP on CF-EDGE-1 can match nothing

Pinned in tests/gauntlet/replays-flowspec.yaml.

control: a packet length a packet can have is unknown, not pass

now: unknown

On the reconstruction, hammerhead preflight on a before and an after snapshot returned unknown.

Text no reason may contain

  • Flowspec

Blockers pinned (some blocker contains each line)

  • flowspec_effect_unmodeled
  • CF-EDGE-1
  • packet-length 512-1500

Pinned in tests/gauntlet/replays-flowspec.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Flowspec parsed and checked structurally

Junos flow routes were a lexer error and IOS-XR flowspec an unparsed block. A rule that starves BGP or can match nothing fails; any other new rule is unknown.

Found by: CenturyLink, Cloudflare 2013

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The crash is a vendor bug no model predicts, but the rule itself matches an impossible packet length (above any MTU and the 16-bit IP length field), which a semantic lint of the candidate rule can refuse.

Not fixed: BGP Flowspec's effect on forwarding

Statically configured rules are parsed on Junos and IOS-XR and checked structurally (a rule that starves BGP or can match nothing fails), but no rule is applied to a traced flow, so any other new or changed rule is unknown. Rules injected over BGP by a DDoS system, as CenturyLink's was, are in no device config at all.

Recorded against: CenturyLink, Cloudflare 2013

Imported from docs/replays/cloudflare-flowspec-2013/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.