INCIDENT REPLAY · 2015-04-10
Google Compute Engine us-central1: traffic sent to a router running a test configuration
During routine planned maintenance, a miscommunication caused production traffic to be sent to a datacenter router that was running a test configuration. A link saturated and us-central1 saw 18 to 47% packet loss for 14 minutes until the router was disabled.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Google Cloud
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.
Sources
What the public record says
- During routine planned maintenance, a miscommunication caused production traffic to be sent to a datacenter router that was running a test configuration. A link saturated and us-central1 saw 18 to 47% packet loss for 14 minutes until the router was disabled.
- Who: Google Cloud
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- Google said production traffic was sent to a router running a test configuration and a link saturated. We model two production routers and a third attached at a high IGP cost, and a drain that raises the cost of both production routers instead of one.
- The capacity rows declare the test router's links at 10 Gb/s, every other link at 40 Gb/s and 20 Gb/s into the datacenter. Google gave no figure: they are illustrative, and the declaration says so.
- Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
Hammerhead's own first-versus-now record
- First verdict
- pass. Every flow is still delivered and no AS boundary is crossed.
- Verdict now
- fail: RTR-TEST-1 carries no traffic between other devices today and would after the change. The saturation itself is load and is not in any config; with capacity and demand DECLARED (
--link-capacity, illustrative figures intests/gauntlet/gce_2015/capacity/, rows inreplays-capacity.yaml) the pushed change fails on the load itself: 20 Gb/s on each test-router link against 10, and the intended drain passes.
Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.
as intended, capacity declared: one production router carries all 20 Gb/s within 40
now: passOn the reconstruction, hammerhead preflight on a before and an after snapshot with --link-capacity @gce_2015/capacity/link_capacity.yaml returned pass.
Pinned in tests/gauntlet/replays-capacity.yaml.
as done, capacity declared, path changes waived: the test router's links are overloaded
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot with --link-capacity @gce_2015/capacity/link_capacity.yaml --allow-path-changes returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- link capacity: GigabitEthernet0/0 on RTR-TEST-1 would carry 20 Gb/s inbound against 10 Gb/s declared, up from 0 Mb/s
- link capacity: GigabitEthernet0/1 on RTR-TEST-1 would carry 20 Gb/s outbound against 10 Gb/s declared, up from 0 Mb/s
Pinned in tests/gauntlet/replays-capacity.yaml.
as intended: one production router drained onto the other
now: passOn the reconstruction, hammerhead preflight on a before and an after snapshot returned pass.
Pinned in tests/gauntlet/replays-individual.yaml.
as done: traffic moved off both
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- path change: RTR-TEST-1 carries no traffic between other devices today and would carry it
Pinned in tests/gauntlet/replays-individual.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
A router drawn into the path
The AS-level path check is blind inside one AS. A device present in both snapshots, in the middle of no path today and of some path after, is named. Every ECMP branch is walked, because the single-path walker follows one member and the others are not idle (the Azure drain caught the first version of this).
Found by: GCE 2015, AWS EBS 2011, T-Mobile
Declared link capacity and shared-risk groups (--link-capacity, --srlg)
A router drawn into the path could be named but not judged: no config says what a link carries. With capacity and demand declared, a link over capacity that was not before fails; a declared group is failed as a unit in both snapshots.
Found by: T-Mobile, GCE 2015, Cloudflare 2020
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as in scope: A snapshot that includes the router's actual (test) config plus the maintenance change would show traffic converging on a path that cannot carry it.
Not fixed: Load, and what a router is built to carry
No config says. The finding is that traffic moves onto a router that carries nothing today; whether it can is the operator's to confirm. With capacity and demand declared (--link-capacity), the capacity rows fail on load for T-Mobile, GCE 2015 and Cloudflare 2020, with illustrative figures, since no public record gives them. Load under a link failure, queueing and router CPU are still not modelled.
Recorded against: GCE 2015, AWS EBS 2011, T-Mobile, Google Cloud Aug 2016
Imported from docs/replays/gce-2015/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.