HomeIncident library
Incident library

INCIDENT REPLAY · 2023-11-22

AMS-IX 'LACP leakage' outage

The Amsterdam peering platform was degraded for more than four hours; traffic fell from roughly 10 Tbps to about 2.1 Tbps. AMS-IX presented the post-mortem at RIPE 87.

Layer 2L2 loop / spanning treeCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Amsterdam Internet Exchange (AMS-IX)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • The Amsterdam peering platform was degraded for more than four hours; traffic fell from roughly 10 Tbps to about 2.1 Tbps. AMS-IX presented the post-mortem at RIPE 87.
  • Who: Amsterdam Internet Exchange (AMS-IX)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch.
  • A newly provisioned port with no effective layer-2 protocol filter; the cascade depended on vendor forwarding behaviour that is not modelled. Modelled as the new port in the peering VLAN with neither BPDU guard nor storm control.
  • No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
  • A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

no change: an access port with no BPDU guard

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.interfaces.bpdu_guard_on_access. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • IX-SW-1
  • GigabitEthernet1/0/17

First run, before the fixes this batch prompted

first: skipped-stub (the check was a stub)

Need BPDU-guard attribute.

Pinned in tests/gauntlet/replays-l2.yaml.

no change: an access port with no storm control

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.interfaces.storm_control_enabled. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • IX-SW-1
  • GigabitEthernet1/0/17

First run, before the fixes this batch prompted

first: skipped-stub (the check was a stub)

Need storm-control projection.

Pinned in tests/gauntlet/replays-l2.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Layer-2 protection parsed and audited

spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.

Found by: Layer-2 family

IOS global portfast / BPDU-guard defaults

Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.

Found by: Layer-2 family

Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR

The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.

Found by: Layer-2 family, AS 7007

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: A missing or ineffective L2 protocol filter on a newly provisioned port is a config-audit finding, but the cascade depended on vendor forwarding behaviour and an RSVP software bug that no config model would predict.

Imported from docs/replays/ams-ix-2023/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.