INCIDENT REPLAY · 2018-10-11

Google Cloud us-central1-c: BGP sessions deleted during router replacement rollback

While replacing the routers that link us-central1-c to Google's backbone, a rollback workflow pushed a configuration meant for the new routers onto the older routers. That deleted their BGP sessions, and the zone lost connectivity to the backbone for 41 minutes. 61% of VMs in the zone lost external connectivity.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Google Cloud

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

https://status.cloud.google.com/incident/cloud-networking/18016

What the public record says

While replacing the routers that link us-central1-c to Google's backbone, a rollback workflow pushed a configuration meant for the new routers onto the older routers. That deleted their BGP sessions, and the zone lost connectivity to the backbone for 41 minutes. 61% of VMs in the zone lost external connectivity.
Who: Google Cloud

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

The rollback is modelled as the old routers losing their BGP neighbour statements toward the backbone, which is what the post-mortem says the wrong configuration did.
Two old routers, the backbone as one router, the zone's fabric as one router.
Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict: fail, for the cause: two BGP adjacencies lost between devices still present, and the zone's subnet lost from the backbone.
Verdict now: unchanged.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the zone's routers and the backbone

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

routing adjacencies: 2 lost
connected subnet(s) reachable before are not reachable after

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
Why the catalog counts this outage as in scope: The pushed config removed every BGP session on the zone's backbone-facing routers; a pre-push diff that simulates the candidate would show all sessions gone and the zone unreachable.

Imported from docs/replays/google-cloud-2018/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.