INCIDENT REPLAY · 2012-01 (blog post date; incident date not confirmed)

Etsy global outage from multicast on unprepared switches

Etsy tried multicast-based replication of Solr search indexes; the multicast traffic flooded the production network and caused a site-wide outage, after which Etsy moved to BitTorrent for index distribution.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Etsy

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account low.

Sources

What the public record says

Etsy tried multicast-based replication of Solr search indexes; the multicast traffic flooded the production network and caused a site-wide outage, after which Etsy moved to BitTorrent for index distribution.
Who: Etsy

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

The family template (tools/replays/gen_family_l2.py): switches in Cisco IOS syntax carrying VLAN 100, with every port protected (portfast, BPDU guard, storm control) except the ones the record points at, so that each finding is the incident's port or switch.
A new application's multicast traffic flooded switches that were not prepared for it. The trigger is application behaviour; the missing control is config-visible: modelled as one switch with IGMP snooping turned off among switches that have it on.
No change is replayed. What started the loop or the storm was physical or an application; the audit runs on the healthy configuration and reports the missing protection. The loop itself, and spanning tree flapping under a BPDU flood, are dynamics no steady-state model has.
A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

no change: IGMP snooping turned off

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.security.igmp_snooping_disabled. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

1 row(s)
RACK-SW-3

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-l2.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Layer-2 protection parsed and audited

spanning-tree portfast|bpduguard|bpdufilter|guard, storm-control and ip igmp snooping were unparsed on IOS. Interface and Vlan nodes carry them; six audits (four were stubs) read them.

Found by: Layer-2 family

IOS global portfast / BPDU-guard defaults

Unparsed: the audit called every globally guarded port unguarded, and deleting the global line passed.