INCIDENT REPLAY · 2015-05-13
AMS-IX peering LAN loop during 100GE module testing
At 12:20 CEST, while engineers tested newly installed 100GE modules ahead of planned maintenance, four 100GE ports were looped while still members of the production peering VPLS. Broadcast traffic over the loop made member routers' MAC addresses appear behind the looped ports, line-card CPUs spiked, and about 500 of 600 route-server BGP sessions (74.5% of sessions platform-wide) dropped. The NOC shut the ports at 12:29 and most sessions were back by 12:40.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Amsterdam Internet Exchange (AMS-IX)
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.
Sources
- https://atnog.at/pipermail/atnog/2015-May/000039.html
- https://issues.edpnet.be/?p=709
- https://labs.ripe.net/author/emileaben/does-the-internet-route-around-damage-a-case-study-using-ripe-atlas/
- https://nltimes.nl/2015/05/13/outage-amsterdam-internet-hub-affects-much-netherlands
- https://www.ams-ix.net/ams/news/outage-at-the-ams-ix-platform-in-amsterdam
- https://www.datacenterdynamics.com/en/news/amsterdam-internet-exchange-experiences-partial-outage/
- https://www.telecompaper.com/news/ams-ix-promises-better-communication-after-service-outage--1083599
What the public record says
- At 12:20 CEST, while engineers tested newly installed 100GE modules ahead of planned maintenance, four 100GE ports were looped while still members of the production peering VPLS. Broadcast traffic over the loop made member routers' MAC addresses appear behind the looped ports, line-card CPUs spiked, and about 500 of 600 route-server BGP sessions (74.5% of sessions platform-wide) dropped. The NOC shut the ports at 12:29 and most sessions were back by 12:40.
- Who: Amsterdam Internet Exchange (AMS-IX)
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The family template (
tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch. - A physical loopback on a port under test. The config-visible precondition is that the test port was still in the production VLAN with no protection: modelled as an access port in the peering VLAN with no BPDU guard, next to customer ports that have it.
- No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
- A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
no change: an access port with no BPDU guard
now: fail (finding)An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.interfaces.bpdu_guard_on_access. Pinned status: fail, a finding whose evidence contains the text below.
Evidence pinned (the evidence contains each line)
- 1 row(s)
- IX-SW-1
- GigabitEthernet1/0/9
First run, before the fixes this batch prompted
first: skipped-stub (the check was a stub)Need BPDU-guard attribute.
Pinned in tests/gauntlet/replays-l2.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
Layer-2 protection parsed and audited
spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.
Found by: Layer-2 family
IOS global portfast / BPDU-guard defaults
Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.
Found by: Layer-2 family
Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR
The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.
Found by: Layer-2 family, AS 7007
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as partly in scope: The physical loopback is not in any config, but 'test ports still bound to the production VPLS/VLAN with no L2 ACL' is a config-visible precondition a pre-maintenance check can refuse.
Imported from docs/replays/ams-ix-2015/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.