HomeIncident library
Incident library

INCIDENT REPLAY · 2017-11-06

Level 3 BGP route leak (Comcast and other US ISPs disrupted)

At 17:47 UTC Level 3 began globally announcing thousands of BGP routes learned from customers and peers that were meant to stay internal to its network, per Oracle/Dyn's analysis. Traffic for large eyeball networks such as Comcast and Bell Canada and for content providers such as Netflix was drawn through Level 3's misconfigured routers, causing a nationwide slowdown and outage for Comcast users and others. Many leaked routes stayed in circulation until about 19:24 UTC, roughly 90 minutes.

Fits no familyBGP leak / hijackCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Level 3 Communications (AS3356); affected networks included Comcast, Bell Canada, Spectrum, Verizon, Cox, RCN and Netflix

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • At 17:47 UTC Level 3 began globally announcing thousands of BGP routes learned from customers and peers that were meant to stay internal to its network, per Oracle/Dyn's analysis. Traffic for large eyeball networks such as Comcast and Bell Canada and for content providers such as Netflix was drawn through Level 3's misconfigured routers, causing a nationwide slowdown and outage for Comcast users and others. Many leaked routes stayed in circulation until about 19:24 UTC, roughly 90 minutes.
  • Who: Level 3 Communications (AS3356); affected networks included Comcast, Bell Canada, Spectrum, Verizon, Cox, RCN and Netflix

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Level 3 did not publish the change. We assume the export prefix-list toward a peer went from the customer's aggregate to the aggregate le 24, letting 48 traffic-engineering /24s out.
  • Level 3's routers and the customer are in the snapshot; the peers are not, which is the operator's seat.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
pass. New announcements were only ever a caveat.
Verdict now
fail: the announcement to the peer goes from 1 prefix to 49 in one change (the announcement-growth check added earlier in this batch).

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

Level 3 and its customer; the peers are outside the snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • L3-EDGE-1 would announce 49 prefixes to
  • up from 1

Pinned in tests/gauntlet/replays-individual.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

An announcement that multiplies

The announcement to an external peer goes from a few prefixes to many in one change.

Found by: Mis-origination family, Level 3

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A change that causes internal or peer-learned more-specific routes to be exported to the global table is detectable as an export-policy / announcement diff before the push.

Imported from docs/replays/level3-2017/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.