INCIDENT REPLAY · 2023-06-29
GitHub: secondary edge path misconfiguration exposed by failover test
GitHub ran a live failover test of a second Internet edge facility. A network path configuration problem on the secondary side meant traffic could not be carried properly, and GitHub was unavailable for much of North and South America for about 32 minutes.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: GitHub
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.
Sources
What the public record says
- GitHub ran a live failover test of a second Internet edge facility. A network path configuration problem on the secondary side meant traffic could not be carried properly, and GitHub was unavailable for much of North and South America for about 32 minutes.
- Who: GitHub
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- GitHub said a network path configuration problem on the secondary side was exposed by a live failover test. We assume the IGP was never enabled on the link between the second edge and the core, which nothing noticed while no traffic used it.
- The change replayed is the failover itself (the primary edge stops announcing).
- Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
Hammerhead's own first-versus-now record
- First verdict
- fail, for the cause: the front-end subnet is lost from the transit.
- Verdict now
- unchanged. Run BEFORE the test this is the insurance case: the failover is checked on the model instead of on customers.
Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.
both edges, the core and a transit
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- connected subnet(s) reachable before are not reachable after
- 198.18.0.0/24 on CORE-1
Pinned in tests/gauntlet/replays-individual.yaml.
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as in scope: A latent broken backup path is precisely what failure simulation on the existing configs reveals before anyone drains the primary.
Imported from docs/replays/github-2023/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.