INCIDENT REPLAY · 2014-08-25
Stack Exchange iptables change blocks load-balancer return traffic
All Stack Exchange sites were down for about six minutes (19:26-19:32 UTC). A change to the iptables rules on the HAProxy load balancers, pushed from Git by Puppet, caused SYN/ACK replies from the IIS web tier to be dropped, so the load balancers could no longer complete connections to any backend.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Stack Exchange / Stack Overflow
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.
Sources
What the public record says
- All Stack Exchange sites were down for about six minutes (19:26-19:32 UTC). A change to the iptables rules on the HAProxy load balancers, pushed from Git by Puppet, caused SYN/ACK replies from the IIS web tier to be dropped, so the load balancers could no longer complete connections to any backend.
- Who: Stack Exchange / Stack Overflow
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The ruleset is ours: an INPUT chain with policy DROP, loopback, established/related, 80/443, management SSH and ICMP accepts, and the change deletes the established/related line. The post-mortem says the pushed rules dropped SYN/ACK replies from the web tier, which is what deleting that line does.
- One host. An iptables-save file has no interfaces and no routes, so there is no address for the sweep to probe.
- Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
Hammerhead's own first-versus-now record
- First verdict
- fail, but only on the generic ACL-change tripwire (five line changes, 'flow disposition may differ'). With
--allow-flow-changes: pass, and the ledger called a sweep that compared nothingexhaustive. - Verdict now
- fail, named: the chain no longer accepts established connections and ends in deny. With the waiver: unknown, because the sweep had nothing to probe and now says so instead of calling that exhaustive.
Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.
the load balancer's iptables ruleset
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- stateful filter: iptables-host filter-INPUT no longer accepts established connections, and the chain ends in deny
Pinned in tests/gauntlet/replays-individual.yaml.
the same, with flow changes waived: the sweep has nothing to probe
now: unknownOn the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned unknown.
Caveats pinned (some caveat contains each line; a caveat never moves the verdict)
- The reachability sweep compared nothing
Pinned in tests/gauntlet/replays-individual.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
A deny chain that stops accepting established connections
An iptables host has no address to probe. The filter is read instead.
Found by: Stack Exchange
A sweep that probed nothing is nothing-to-probe, not exhaustive
The ledger called 0 pairs exhaustive; the CLI banner had no words for it and a debug build panicked. unknown now, with the caveat in words.
Found by: Stack Exchange
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as in scope: iptables is a parsed vendor; a candidate ruleset that drops return traffic from the web tier to the load balancer is a stateful reachability regression on a flow that was permitted before the change.
Green links. Broken checkout.
Chicago payments network. Four routed metro nodes form a ring; the service gateway has a separate handoff. Route checks and ICMP succeed. The primary service transport fails even though an allow rule appears in the service access list. Investigate the two service networks from R2: 10.201.130.0/24 to 10.201.136.0/24 (TCP/443), and 10.201.131.0/24 to 10.201.137.0/24 (TCP/8443).
Link addressing (5)
- R1 R2
10.201.144.0/30 - R2 R3
10.201.145.0/30 - R3 R4
10.201.146.0/30 - R4 R1
10.201.147.0/30 - R4 R5
10.201.148.0/30
Free to practice. No account required.
Imported from docs/replays/stack-exchange-2014/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.