HomeIncident library
Incident library

INCIDENT REPLAY · 2016-04-11

Google Compute Engine: every IP block withdrawn

Engineers retired one unused IP block; the management software pushed a configuration with every block removed, and every site stopped announcing them.

Hand-built replaysNot classified (hand-built)Catalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

The summary at the top is Hammerhead's own one-line description of the replay; the sources below are the record. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • "On Monday, 11 April, 2016, Google Compute Engine instances in all regions lost external connectivity for a total of 18 minutes, from 19:09 to 19:27 Pacific Time."
  • The intended change: "At 14:50 Pacific Time on April 11th, our engineers removed an unused GCE IP block from our network configuration".
  • What was pushed instead: "the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network."
  • The effect on the network: "those sites which had been announcing GCE IP blocks ceased to do so when they received the new configuration."
  • A safeguard saw it and was not heard: "the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step's conclusion back to the push process".
  • Root cause, in Google's words: "a software bug in the network configuration management software."

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours. Google has published none of its configuration.

  • Two peering sites and one compute gateway stand in for Google's network. Four /20 blocks stand in for the GCE IP blocks: three in use, one unused. All addresses are from 198.18.0.0/15 and the documentation ranges; AS 64496 stands in for Google and AS 64500 / AS 64501 for two transit networks that are outside the snapshot, as the rest of the internet is for any operator.
  • Cisco IOS syntax. Each site announces a block with a network statement backed by a static to Null0. The instances' subnets are connected on the compute gateway and carried in OSPF, so internal reachability does not depend on the announcements, which is what the report describes: the instances were up and unreachable from outside.
  • after_intended removes the unused block from both sites. after_pushed removes every block from both sites. The report says the configuration propagated site by site with the last site withdrawing at 19:09; we model only the end state.
  • The bugs were in management software, which Hammerhead does not see. What it sees is the configuration that software produced, before it is pushed. That is the claim this replay tests, and the only one.

tools/replays/gen_gce_2016.py writes the corpus.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
The push: fail. But the intended change failed too, and under the only waiver there was (--allow-advertisement-changes) the push that withdrew everything was pass.
Verdict now
Intended change with --allow-withdrawal 198.18.48.0/20: pass. The push with the same flag: fail, "would stop announcing 3 prefix(es)", the three nobody named.
Fixed?
Yes. A scoped waiver. The blanket one is how a small intended withdrawal becomes all of them behind a gate that was told to expect a withdrawal.

Quoted from docs/replays/README.md. “First” is the tool before the fixes this replay prompted.

This replay's runs are pinned in Hammerhead's hand-written Rust tests rather than a manifest, so the record above is the only account quoted here.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

--allow-withdrawal <PREFIX>

The advertisement gate's only waiver was blanket, so expecting one withdrawal disarmed it for all of them.

Found by: Google Compute Engine

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.

Imported from docs/replays/gce-2016/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.