HomeIncident library
Incident library

INCIDENT REPLAY · 2011-04-21

AWS EC2/EBS us-east-1: traffic shifted onto low-capacity redundant network

During a change to upgrade capacity on the primary EBS network in one Availability Zone, traffic was shifted off a primary router incorrectly. Instead of going to the other primary router it went onto the lower-capacity redundant EBS network, isolating many EBS nodes; when connectivity returned a re-mirroring storm left 13% of volumes in the AZ stuck. Full recovery took about 3 days.

Fits no familyCapacity / traffic shift after changeCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Amazon Web Services

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • During a change to upgrade capacity on the primary EBS network in one Availability Zone, traffic was shifted off a primary router incorrectly. Instead of going to the other primary router it went onto the lower-capacity redundant EBS network, isolating many EBS nodes; when connectivity returned a re-mirroring storm left 13% of volumes in the AZ stuck. Full recovery took about 3 days.
  • Who: Amazon Web Services

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Same shape as GCE 2015, and the same template: the redundant lower-capacity network is a router attached at a high IGP cost, and the traffic shift takes both primary routers out of the path instead of one.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
pass.
Verdict now
fail: SECONDARY-NET-1 carries nothing today and would carry everything. The re-mirroring storm that followed is application behaviour and out of remit.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

as intended: one production router drained onto the other

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot returned pass.

Pinned in tests/gauntlet/replays-individual.yaml.

as done: traffic moved off both

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • path change: SECONDARY-NET-1 carries no traffic between other devices today and would carry it

Pinned in tests/gauntlet/replays-individual.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

A router drawn into the path

The AS-level path check is blind inside one AS. A device present in both snapshots, in the middle of no path today and of some path after, is named. Every ECMP branch is walked, because the single-path walker follows one member and the others are not idle (the Azure drain caught the first version of this).

Found by: GCE 2015, AWS EBS 2011, T-Mobile

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A pre-push simulation of the traffic shift shows EBS traffic converging on a network that cannot carry it; this is the textbook 'shifted onto a path that cannot carry it' case.

Not fixed: Load, and what a router is built to carry

No config says. The finding is that traffic moves onto a router that carries nothing today; whether it can is the operator's to confirm. With capacity and demand declared (--link-capacity), the capacity rows fail on load for T-Mobile, GCE 2015 and Cloudflare 2020, with illustrative figures, since no public record gives them. Load under a link failure, queueing and router CPU are still not modelled.

Recorded against: GCE 2015, AWS EBS 2011, T-Mobile, Google Cloud Aug 2016

Imported from docs/replays/aws-ebs-2011/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.