INCIDENT REPLAY · 2021-03-24
Georgia Tech: building VLAN spanned into the Coda data centre loops
For about an hour (roughly 14:30-15:30) users could not reach research clusters in the Coda data centre, and connectivity problems were widespread across campus as core network equipment was affected.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Georgia Institute of Technology (OIT; PACE research computing)
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.
Sources
What the public record says
- For about an hour (roughly 14:30-15:30) users could not reach research clusters in the Coda data centre, and connectivity problems were widespread across campus as core network equipment was affected.
- Who: Georgia Institute of Technology (OIT; PACE research computing)
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The family template (
tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch. - A building VLAN stretched into the data-centre core. Modelled as VLAN 100 carried from a building's access switches across the campus distribution into the data-centre switches, eight switches in a chain; the patch that looped it is physical. The trunks carry no addresses, so the check's number is the switch count standing in as the upper bound on the diameter, and the row says so.
- No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
- A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
no change: a VLAN whose spanning tree is wider than seven bridges
now: fail (finding)An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.topology.vlan_spans_many_switches. Pinned status: fail, a finding whose evidence contains the text below.
Evidence pinned (the evidence contains each line)
- 1 row(s)
- 100
Pinned in tests/gauntlet/replays-l2.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
Layer-2 protection parsed and audited
spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.
Found by: Layer-2 family
Measured STP diameter, rings, BPDU-filtered loops
The domain-size audit counted switches; it now reads the measured 802.1D diameter where the known links join every switch, and the switch count as a marked upper bound where they do not. A ring with a BPDU-filtered trunk is a forwarding loop.
Found by: Beth Israel, Georgia Tech 2021, Fog Creek
IOS global portfast / BPDU-guard defaults
Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.
Found by: Layer-2 family
Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR
The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.
Found by: Layer-2 family, AS 7007
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as partly in scope: A VLAN stretched between a campus building and the data-centre core is a config-visible failure-domain finding even if the looping patch itself is physical.
Not fixed: Spanning tree over links the model does not know
L2 adjacency comes from L3 links, so over unaddressed trunks a ring or loop is NULL, not false, and the domain size is the switch count, marked as an upper bound.
Recorded against: Beth Israel, Georgia Tech 2021
Imported from docs/replays/georgia-tech-2021/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.