HomeIncident library
Incident library

INCIDENT REPLAY · 2016-08-05

Google Cloud: GCP prefixes announced from a single PoP during router drain

While removing a faulty router from service, engineers used a new traffic-diversion procedure. The configuration it applied caused some GCP IP ranges to be announced from a single point of presence in the southwestern US, pulling traffic into infrastructure with overly restrictive filters. Non-TCP/UDP traffic (ICMP, ESP, AH, SCTP) was dropped, Cloud VPN tunnels lost all data traffic and Cloud Router BGP sessions dropped for 99 minutes.

Fits no familyRouting policy errorCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Google Cloud

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • While removing a faulty router from service, engineers used a new traffic-diversion procedure. The configuration it applied caused some GCP IP ranges to be announced from a single point of presence in the southwestern US, pulling traffic into infrastructure with overly restrictive filters. Non-TCP/UDP traffic (ICMP, ESP, AH, SCTP) was dropped, Cloud VPN tunnels lost all data traffic and Cloud Router BGP sessions dropped for 99 minutes.
  • Who: Google Cloud

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Three points of presence announce the same two blocks. The intended change drains one; the procedure as run took two of the three out, leaving one to take all inbound traffic.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
not runnable as the operator would run it: the only waivers were the blanket flag and (later) the bare prefix, and either one waves the same withdrawal through at every site.
Verdict now
pass for the drain with --allow-withdrawal PREFIX@GCP-POP-ATL, fail for the push: GCP-POP-DFW was not named. The congestion at the last site is load, which no config contains; the withdrawal is the cause and that is what fails.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the change as intended, each withdrawal named for its router

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-withdrawal 198.18.0.0/20@GCP-POP-ATL --allow-withdrawal 198.18.16.0/20@GCP-POP-ATL returned pass.

Pinned in tests/gauntlet/replays-individual.yaml.

the change as pushed, from the operator's own routers

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-withdrawal 198.18.0.0/20@GCP-POP-ATL --allow-withdrawal 198.18.16.0/20@GCP-POP-ATL returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • GCP-POP-DFW would stop announcing

Text no reason may contain

  • GCP-POP-ATL would stop announcing

Pinned in tests/gauntlet/replays-individual.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

--allow-withdrawal PREFIX@DEVICE

Draining one of several routers that announce a prefix is a withdrawal there only; the bare prefix waved it through everywhere.

Found by: Google Cloud Aug 2016

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: Both halves are config-visible: the change altered where prefixes are originated, and the path it attracted traffic onto carried filters that drop ESP/ICMP; a before/after reachability sweep by protocol would flag it.

Not fixed: Load, and what a router is built to carry

No config says. The finding is that traffic moves onto a router that carries nothing today; whether it can is the operator's to confirm. With capacity and demand declared (--link-capacity), the capacity rows fail on load for T-Mobile, GCE 2015 and Cloudflare 2020, with illustrative figures, since no public record gives them. Load under a link failure, queueing and router CPU are still not modelled.

Recorded against: GCE 2015, AWS EBS 2011, T-Mobile, Google Cloud Aug 2016

Imported from docs/replays/google-cloud-2016-08/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.