HomeIncident library
Incident library

INCIDENT REPLAY · 2013-12-17

AWS sa-east-1: bad config on restarted network device advertised invalid routes

Utility power loss and generator failures took down part of one AZ in Sao Paulo. During recovery a technician manually restarted a network device and introduced a bad configuration, so it advertised invalid routes and degraded internet connectivity for both AZs in the region for about 20 minutes.

Fits no familyRouting policy errorCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Amazon Web Services

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • Utility power loss and generator failures took down part of one AZ in Sao Paulo. During recovery a technician manually restarted a network device and introduced a bad configuration, so it advertised invalid routes and degraded internet connectivity for both AZs in the region for about 20 minutes.
  • Who: Amazon Web Services

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • AWS said the restarted device advertised invalid routes and did not say which. We assume a static discard for half of another device's instance subnet, redistributed into the IGP.
  • We chose the half the owner's interface address is NOT in, on purpose: that is the case a one-address-per-subnet probe misses.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
pass. The sweep probed the owner's address in the subnet, which the stray route does not cover.
Verdict now
fail: 198.18.1.128/25 inside 198.18.1.0/24 is lost from every other device. The sweep now probes inside every route a change adds for part of a connected subnet.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

one availability zone's cores and aggregation

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • 198.18.1.128/25

Pinned in tests/gauntlet/replays-individual.yaml.

one availability zone's cores and aggregation (potential microloop)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail.

This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

  • AZ-AGG-2 -> AZ-CORE-1 -> AZ-AGG-2, if AZ-AGG-2 has its new next hop while AZ-CORE-1 still has its old one

Pinned in tests/gauntlet/replays-transients.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Probes inside a route that carves up a connected subnet

The sweep probed one address per subnet, the owner's. A stray route for the other half left it alone.

Found by: AWS sa-east-1

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: The network half is a hand-entered device config that originates invalid routes; verifying the candidate before it is activated would show the bad advertisements. The power half is out of scope.

Imported from docs/replays/aws-sa-east-1-2013/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.