HomeIncident library
Incident library

INCIDENT REPLAY · 2007-08-11

LAX Customs and Border Protection LAN outage

CBP's local network at LAX slowed from ~12:50 PDT and stopped passing database queries at 16:16; full processing resumed at 23:40. More than 17,000 arriving international passengers were held, about 60 aircraft sat on the tarmac for hours, and some flights diverted to Ontario, CA. Technicians eventually restored service by disconnecting a circuit carrying 12 devices in the Bradley terminal.

Layer 2L2 loop / spanning treeCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: US Customs and Border Protection at Los Angeles International Airport; Sprint (WAN vendor)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • CBP's local network at LAX slowed from ~12:50 PDT and stopped passing database queries at 16:16; full processing resumed at 23:40. More than 17,000 arriving international passengers were held, about 60 aircraft sat on the tarmac for hours, and some flights diverted to Ontario, CA. Technicians eventually restored service by disconnecting a circuit carrying 12 devices in the Bradley terminal.
  • Who: US Customs and Border Protection at Los Angeles International Airport; Sprint (WAN vendor)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch.
  • A failing NIC is hardware. What let it take the LAN down is one flat broadcast domain with no storm control on the port it sat on: modelled as an access port with BPDU guard and portfast but no storm control, on a switch whose other ports have all three.
  • No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
  • A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

no change: an access port with no storm control

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.interfaces.storm_control_enabled. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • TERMINAL-SW-1
  • GigabitEthernet1/0/2

First run, before the fixes this batch prompted

first: skipped-stub (the check was a stub)

Need storm-control projection.

Pinned in tests/gauntlet/replays-l2.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Layer-2 protection parsed and audited

spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.

Found by: Layer-2 family

IOS global portfast / BPDU-guard defaults

Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.

Found by: Layer-2 family

Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR

The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.

Found by: Layer-2 family, AS 7007

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The trigger was a failing NIC (hardware), but a single flat broadcast domain spanning all terminals with no storm-control or segmentation is visible in switch configs and is the reason one host could take down the port of entry.

Imported from docs/replays/lax-cbp-2007/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.