HomeIncident library
Incident library

INCIDENT REPLAY · 2014-08-27

Time Warner Cable nationwide Internet outage

At about 4:30 AM ET, during overnight network maintenance, an erroneous configuration was propagated across TWC's national backbone. RIPE Labs observed severe BGP prefix withdrawals across most of TWC's seven ASNs from roughly 09:30 to 11:00 UTC, with 0-2% reachability for several of them and at least 64 single-homed downstream networks taken offline. Internet and on-demand video were knocked out in all TWC markets, affecting on the order of 11 million residential broadband customers, with service largely restored by about 6:00 AM ET (TWC later said 7:30 AM).

Fits no familyOtherCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Time Warner Cable (AS7843 backbone, plus AS10796, AS11351, AS11426, AS11427, AS12271, AS20001; now Charter/Spectrum)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • At about 4:30 AM ET, during overnight network maintenance, an erroneous configuration was propagated across TWC's national backbone. RIPE Labs observed severe BGP prefix withdrawals across most of TWC's seven ASNs from roughly 09:30 to 11:00 UTC, with 0-2% reachability for several of them and at least 64 single-homed downstream networks taken offline. Internet and on-demand video were knocked out in all TWC markets, affecting on the order of 11 million residential broadband customers, with service largely restored by about 6:00 AM ET (TWC later said 7:30 AM).
  • Who: Time Warner Cable (AS7843 backbone, plus AS10796, AS11351, AS11426, AS11427, AS12271, AS20001; now Charter/Spectrum)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • What the erroneous configuration was has not been published. We assume it removed the backbone's originations, because what outside observers measured was the prefixes leaving the global table.
  • Two edge routers, one core, one access network, two transits.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
fail, for the cause: from the operator's own routers the advertisement gate names the six withdrawals; with the transits in the snapshot the subscriber subnet is lost from both.
Verdict now
unchanged.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the backbone and its transits in one snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • connected subnet(s) reachable before are not reachable after
  • from TRANSIT-A, TRANSIT-B

Pinned in tests/gauntlet/replays-individual.yaml.

the backbone's own routers, which is what the operator has

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • TWC-EDGE-EAST would stop announcing 3 prefix(es)
  • TWC-EDGE-WEST would stop announcing 3 prefix(es)

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A backbone-wide configuration push during a maintenance window that caused most of the operator's prefixes to be withdrawn from the global table is the announcements-withdrawn / lost-reachability outcome a pre-push simulation is built to flag; class is 'other' only because TWC never said which configuration object was wrong.

Imported from docs/replays/twc-2014/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.