HomeIncident library
Incident library

INCIDENT REPLAY · 2002-11-13

Beth Israel Deaconess Medical Center spanning-tree collapse

The hospital's flat, all-Layer-2 campus network collapsed repeatedly over roughly four days (13-18 Nov 2002). Clinicians reverted to paper, lab turnaround went from ~45 minutes to as long as five hours, and the emergency department went on diversion for hours. Cisco flew in Catalyst 6509s and rebuilt the core with Layer-3 routing.

Layer 2L2 loop / spanning treeCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: CareGroup / Beth Israel Deaconess Medical Center, Boston (CIO John Halamka); Cisco CAP team and Callisma consultants in recovery

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • The hospital's flat, all-Layer-2 campus network collapsed repeatedly over roughly four days (13-18 Nov 2002). Clinicians reverted to paper, lab turnaround went from ~45 minutes to as long as five hours, and the emergency department went on diversion for hours. Cisco flew in Catalyst 6509s and rebuilt the core with Layer-3 routing.
  • Who: CareGroup / Beth Israel Deaconess Medical Center, Boston (CIO John Halamka); Cisco CAP team and Callisma consultants in recovery

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch.
  • The record says the spanning-tree domain had grown past its design diameter (more than seven hops) with no layer-3 boundary. Modelled as one VLAN carried by nine switches in a chain, every edge port protected, so the one finding is the size of the domain. The trunks carry no addresses, as real ones do not, so no configuration states which switch is cabled to which: the check reports the switch count as the upper bound on the diameter and says it is not a measurement.
  • No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
  • A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

no change: a VLAN whose spanning tree is wider than seven bridges

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.topology.vlan_spans_many_switches. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • 100

Pinned in tests/gauntlet/replays-l2.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Layer-2 protection parsed and audited

spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.

Found by: Layer-2 family

Measured STP diameter, rings, BPDU-filtered loops

The domain-size audit counted switches; it now reads the measured 802.1D diameter where the known links join every switch, and the switch count as a marked upper bound where they do not. A ring with a BPDU-filtered trunk is a forwarding loop.

Found by: Beth Israel, Georgia Tech 2021, Fog Creek

IOS global portfast / BPDU-guard defaults

Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.

Found by: Layer-2 family

Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR

The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.

Found by: Layer-2 family, AS 7007

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: Not triggered by a push, but a config-derived L2 model can compute spanning-tree diameter (>7 hops), root placement and the absence of any L3 boundary and flag the design as one loop away from campus-wide collapse.

Not fixed: Spanning tree over links the model does not know

L2 adjacency comes from L3 links, so over unaddressed trunks a ring or loop is NULL, not false, and the domain size is the switch count, marked as an upper bound.

Recorded against: Beth Israel, Georgia Tech 2021

Imported from docs/replays/beth-israel-2002/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.