HomeIncident library
Incident library

INCIDENT REPLAY · 2020-06-15

T-Mobile US nationwide voice/text outage (OSPF link-weight misconfiguration)

While integrating new routers in the Southeast, T-Mobile set Open Shortest Path First (OSPF) link weights wrongly on an already-active router that was not built to carry call-signalling traffic. When a leased fiber transport link failed at 12:33 PM EDT, traffic followed the lowest-weight path to that router, which dropped it (the dropped traffic was MPLS), isolating the Atlanta market; a latent IMS registration software flaw then turned that into a nationwide registration storm. The outage lasted over twelve hours; the FCC estimated at least 41% of calls on the network failed, including at least 23,621 failed 911 calls.

Fits no familyIGP / MPLS core configCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: T-Mobile US (AS21928)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • While integrating new routers in the Southeast, T-Mobile set Open Shortest Path First (OSPF) link weights wrongly on an already-active router that was not built to carry call-signalling traffic. When a leased fiber transport link failed at 12:33 PM EDT, traffic followed the lowest-weight path to that router, which dropped it (the dropped traffic was MPLS), isolating the Atlanta market; a latent IMS registration software flaw then turned that into a nationwide registration storm. The outage lasted over twelve hours; the FCC estimated at least 41% of calls on the network failed, including at least 23,621 failed 911 calls.
  • Who: T-Mobile US (AS21928)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • From the FCC staff report: link weights were set wrongly on an active router not built to carry the traffic, and nothing happened until a leased fibre link failed. The weights and topology are ours: a primary path (10 a link), a designed backup (50 a link), and the third router at 1000 a link before the change and 30 after it. 30 was chosen so that with every link up there is no equal-cost tie and the fault is latent, as it was.
  • That the router DROPPED the traffic (it was MPLS) is not modelled, and the IMS registration storm that made it national is software. What is modelled is the traffic moving onto that router.
  • The capacity rows (replays-capacity.yaml) declare SE-AGG-1's links at 1 Gb/s, every other link at 10 Gb/s and 4 Gb/s of signalling from the market. The FCC report gives no such figure: they are illustrative, and the declaration says so.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
pass, from every angle the tool then had: with every link up the change moves nothing.
Verdict now
three runs. The change alone: still pass, which is the honest answer to the question asked. With --check-link-failures: fail, SE-AGG-1 would carry traffic with the leased fibre (or the link behind it) down. With the fibre declared down in both snapshots by an interface_blacklist file: fail, same finding. With link capacity and demand declared (illustrative figures) and path changes waived: fail on the load, 4 Gb/s on each of SE-AGG-1's links against 1; with every link up the same declaration passes, because nothing moves.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

capacity declared, every link up: the fault is latent and nothing moves

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --link-capacity @tmobile_2020/capacity/link_capacity.yaml returned pass.

Pinned in tests/gauntlet/replays-capacity.yaml.

capacity declared, the leased fibre down in both snapshots, path changes waived

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --link-capacity @tmobile_2020/capacity/link_capacity.yaml --allow-path-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • link capacity: GigabitEthernet0/0 on SE-AGG-1 would carry 4 Gb/s inbound against 1 Gb/s declared, up from 0 Mb/s
  • link capacity: GigabitEthernet0/1 on SE-AGG-1 would carry 4 Gb/s outbound against 1 Gb/s declared, up from 0 Mb/s

Pinned in tests/gauntlet/replays-capacity.yaml.

the weight change as it was made, every link up

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot returned pass.

Pinned in tests/gauntlet/replays-individual.yaml.

the same change, asking for every link to be failed in turn

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --check-link-failures returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • under a link failure: SE-AGG-1 carries no traffic between other devices and would carry it after this change
  • ATL-MKT-1 GigabitEthernet0/0 <-> CORE-A-1 GigabitEthernet0/0

Pinned in tests/gauntlet/replays-individual.yaml.

the same change with the leased fibre declared down in both snapshots

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • path change: SE-AGG-1 carries no traffic between other devices today and would carry it

Pinned in tests/gauntlet/replays-individual.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

A router drawn into the path

The AS-level path check is blind inside one AS. A device present in both snapshots, in the middle of no path today and of some path after, is named. Every ECMP branch is walked, because the single-path walker follows one member and the others are not idle (the Azure drain caught the first version of this).

Found by: GCE 2015, AWS EBS 2011, T-Mobile

--check-link-failures

Each link between two modelled devices taken down in BOTH snapshots, and the pair swept again; what the change breaks or moves only then is a finding. Off by default, 32 links, CLI only.

Found by: T-Mobile

Declared link capacity and shared-risk groups (--link-capacity, --srlg)

A router drawn into the path could be named but not judged: no config says what a link carries. With capacity and demand declared, a link over capacity that was not before fails; a declared group is failed as a unit in both snapshots.

Found by: T-Mobile, GCE 2015, Cloudflare 2020

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: An IGP metric change whose effect only appears under a single link failure (traffic shifting onto a router that cannot forward it) is exactly what a pre-push failure-scenario simulation of the proposed config would expose; the IMS software flaw that spread it nationwide is outside remit.

Not fixed: Load, and what a router is built to carry

No config says. The finding is that traffic moves onto a router that carries nothing today; whether it can is the operator's to confirm. With capacity and demand declared (--link-capacity), the capacity rows fail on load for T-Mobile, GCE 2015 and Cloudflare 2020, with illustrative figures, since no public record gives them. Load under a link failure, queueing and router CPU are still not modelled.

Recorded against: GCE 2015, AWS EBS 2011, T-Mobile, Google Cloud Aug 2016

Imported from docs/replays/tmobile-2020/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.