HomeIncident library
Incident library

INCIDENT REPLAY · 2016-07-20

Southwest Airlines router partial failure (2,300 flights cancelled)

At 13:09 on 20 July a router in Southwest's Dallas Love Field data centre failed in a way its backup did not detect, choking hundreds of applications. About 2,300 flights were cancelled over four days and the airline put the cost in the tens of millions of dollars.

RedundancyHardware / power / fiberCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Southwest Airlines

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • At 13:09 on 20 July a router in Southwest's Dallas Love Field data centre failed in a way its backup did not detect, choking hundreds of applications. About 2,300 flights were cancelled over four days and the airline put the cost in the tens of millions of dollars.
  • Who: Southwest Airlines

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Southwest never published the design. We assume the failover the record describes (a backup that did not detect the failure) is the common static shape: a primary default route through one WAN router and a floating backup at distance 250 through another, with nothing tracking whether the primary next hop still answers. Under a grey failure the link stays up, so the primary stays installed and the backup waits.
  • Three Cisco IOS routers. The healthy variant adds an IP SLA probe to the primary WAN router and track 10 on the primary route; nothing else differs.
  • Whether Southwest's failover used static routes, a routing protocol without BFD, or an application-level health check is not known. The audit covers the static shape only; an IGP adjacency with no BFD is not audited here.
  • No change is replayed: nothing in the record says a configuration change started it. The audit runs on the configuration as it stood.
  • Stand-in address space throughout. A reconstruction, not any operator's configuration, and not a claim that the outage would have been prevented.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

DC-CORE-1 as built: the primary default route has no tracking

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.static.floating_backup_behind_untracked_primary. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • DC-CORE-1
  • 0.0.0.0/0 via 172.16.0.2
  • backup via 172.16.1.2 at distance 250

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

the healthy variant: the primary tracks an IP SLA probe to the WAN router

now: pass

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.static.floating_backup_behind_untracked_primary. Pinned status: pass, the check finds nothing.

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The trigger was a grey hardware failure, which is out of scope; whether failover depends only on hard link-down (no BFD, object tracking or route health check) is a config-visible redundancy weakness, but Southwest never published the design.

Not fixed: Static-route tracking on NX-OS and EOS

The floating-backup audit reads IOS track; the NX-OS and EOS parsers discard it, so the audit answers NULL there.

Recorded against: Southwest 2016

Imported from docs/replays/southwest-2016/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.