INCIDENT REPLAY · 2022-04-05
AMS-IX loop from wrong port configuration before planned maintenance (2022 statement)
AMS-IX published a statement dated 6 April 2022 describing a short outage in which 74.5% of all BGP sessions on the Amsterdam platform were interrupted between 10:20 and 10:45. During preparation for a routing test ahead of planned maintenance, an engineer's action put the wrong configuration on a port and created a loop on the production network; it was cleared by disabling the looping ports. Date caveat: the page does not state the incident day (it says members were told the root cause 'yesterday'), so 5 April 2022 is my best estimate; a search summary blended this statement with the very similar 13 May 2015 loop, and a DCD report returned by search (expected 7.3 Tb/s falling to just over 3 Tb/s) could not be opened to confirm which event it covers.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: AMS-IX Amsterdam (route-server AS6777; peering LAN)
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.
Sources
What the public record says
- AMS-IX published a statement dated 6 April 2022 describing a short outage in which 74.5% of all BGP sessions on the Amsterdam platform were interrupted between 10:20 and 10:45. During preparation for a routing test ahead of planned maintenance, an engineer's action put the wrong configuration on a port and created a loop on the production network; it was cleared by disabling the looping ports. Date caveat: the page does not state the incident day (it says members were told the root cause 'yesterday'), so 5 April 2022 is my best estimate; a search summary blended this statement with the very similar 13 May 2015 loop, and a DCD report returned by search (expected 7.3 Tb/s falling to just over 3 Tb/s) could not be opened to confirm which event it covers.
- Who: AMS-IX Amsterdam (route-server AS6777; peering LAN)
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The family template (
tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch. - A wrong port configuration bridged a port into the production layer-2 domain before a planned maintenance. Modelled as a trunk carrying the peering VLAN with BPDU filtering, which is the one line that lets a bridged loop past spanning tree; the loop itself needs the second end, which is physical.
- No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
- A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
no change: a trunk with BPDU filtering, which spanning tree cannot see through
now: fail (finding)An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.interfaces.bpdu_filter_on_trunk. Pinned status: fail, a finding whose evidence contains the text below.
Evidence pinned (the evidence contains each line)
- 1 row(s)
- IX-SW-2
- GigabitEthernet1/0/24
First run, before the fixes this batch prompted
first: no-such-check (the check did not exist yet)Pinned in tests/gauntlet/replays-l2.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
Layer-2 protection parsed and audited
spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.
Found by: Layer-2 family
IOS global portfast / BPDU-guard defaults
Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.
Found by: Layer-2 family
Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR
The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.
Found by: Layer-2 family, AS 7007
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as partly in scope: A wrong port configuration that bridges a port into the production L2 domain is a config diff a tool could flag, but proving a forwarding loop needs an L2/VPLS model of the exchange fabric plus knowledge of the physical cabling.
Imported from docs/replays/ams-ix-2022/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.