HomeIncident library
Incident library

INCIDENT REPLAY · 2012-05-05

Fog Creek network loop during planned switch reconfiguration

During a planned weekend network maintenance (5-6 May 2012) a loop formed between several switches. The resulting BPDU flood caused spanning tree to flap and hosted services were unavailable for about two hours in total.

Fits no familyL2 loop / spanning treeCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Fog Creek Software (FogBugz / Kiln / Trello hosting)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • During a planned weekend network maintenance (5-6 May 2012) a loop formed between several switches. The resulting BPDU flood caused spanning tree to flap and hosted services were unavailable for about two hours in total.
  • Who: Fog Creek Software (FogBugz / Kiln / Trello hosting)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The record says a loop formed between several switches during a planned reconfiguration and BPDUs flooded. What was reconfigured is not published. We model a third trunk closing a ring of three switches on one VLAN, once with spanning tree left alone and once with BPDU filtering on the new link, which is the usual way a loop gets past spanning tree.
  • The BPDU flood itself, and spanning tree flapping under it, are dynamics no steady-state model has.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
the ring under spanning tree: pass (the simulated election blocks one port, which is right). With BPDU filtering: fail, as a refusal: spanning-tree bpdufilter enable was a line the parser did not model, and a new unparsed line is not certifiable.
Verdict now
fail, for the cause: both ends of the new link would start filtering BPDUs on a trunk, so a loop through it is invisible to spanning tree. The line is parsed now (the layer-2 family needed it), and the moment it was, the refusal became a pass; the finding replaced it. Reporting an L2 ring with no blocked port is still not done.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the ring closed with spanning tree left on: the control

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot returned pass.

Pinned in tests/gauntlet/replays-individual.yaml.

the ring closed with BPDU filtering on the new link

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • layer-2 protection: SW-A-1 GigabitEthernet0/1 would start filtering BPDUs on a trunk, so a loop through it is invisible to spanning tree
  • layer-2 protection: SW-B-1 GigabitEthernet0/1 would start filtering BPDUs on a trunk

Pinned in tests/gauntlet/replays-individual.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Switching off a layer-2 protection is a finding

Parsing bpdufilter turned Fog Creek's refusal into a pass: a sound refusal became silence. BPDU filtering on a trunk (or a trunk born with it), BPDU guard or storm control removed, IGMP snooping turned off, under the policy-widening gate.

Found by: Fog Creek

Measured STP diameter, rings, BPDU-filtered loops

The domain-size audit counted switches; it now reads the measured 802.1D diameter where the known links join every switch, and the switch count as a marked upper bound where they do not. A ring with a BPDU-filtered trunk is a forwarding loop.

Found by: Beth Israel, Georgia Tech 2021, Fog Creek

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A planned switch reconfiguration that introduces a forwarding loop or an unintended spanning-tree topology change is detectable by simulating L2 before and after the change.

Imported from docs/replays/fog-creek-2012/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.