INCIDENT REPLAY · 2009-11-19

FAA FTI flight-plan network outage (Salt Lake City router)

A router replaced during maintenance at the FTI hub in Salt Lake City carried a routing error in its configuration. From about 05:00 EST flight-plan (NADIN) and traffic-flow data stopped moving nationally for roughly four hours; controllers entered flight plans by hand and hundreds of flights were delayed. The fault also took down a second node in Hampton, Georgia, and did not fail over.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: FAA; Harris Corp. (FAA Telecommunications Infrastructure contractor)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

A router replaced during maintenance at the FTI hub in Salt Lake City carried a routing error in its configuration. From about 05:00 EST flight-plan (NADIN) and traffic-flow data stopped moving nationally for roughly four hours; controllers entered flight plans by hand and hundreds of flights were delayed. The fault also took down a second node in Hampton, Georgia, and did not fail over.
Who: FAA; Harris Corp. (FAA Telecommunications Infrastructure contractor)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

The record says a replacement router carried a routing error. We assume a static route for the flight-plan subnet pointing at the wrong neighbour, which loops between the two hubs.
Two hubs, the flight-plan gateway behind one, one control centre behind the other.
Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict: fail, for the cause: the flight-plan subnet is lost from every other device.
Verdict now: unchanged.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

two hubs, the flight-plan gateway and a centre

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

connected subnet(s) reachable before are not reachable after
198.18.10.0/24 on NADIN-SLC-1

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
Why the catalog counts this outage as in scope: A replacement router's config differing from the unit it replaces, and the resulting loss of most routes between two routing domains, is a FIB-diff and reachability-regression finding before the swap.

Imported from docs/replays/faa-fti-2009/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.