INCIDENT REPLAY · 2005-05-07
Cogent originates a Google /24 (Google May 2005 outage)
At 14:37 UTC AS174 began originating Google's 64.233.161.0/24. A Carleton University study found 49.1% of the ASes re-advertising that prefix switched to the bogus path. Google's own outage later that day (about 22:10 UTC, 15 to 60 minutes) was publicly attributed by Google to internal DNS misconfiguration, so the link between the two is unproven.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Cogent (AS174) originated 64.233.161.0/24; Google (AS15169) victim
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.
Sources
What the public record says
- At 14:37 UTC AS174 began originating Google's 64.233.161.0/24. A Carleton University study found 49.1% of the ASes re-advertising that prefix switched to the bogus path. Google's own outage later that day (about 22:10 UTC, 15 to 60 minutes) was publicly attributed by Google to internal DNS misconfiguration, so the link between the two is unproven.
- Who: Cogent (AS174) originated 64.233.161.0/24; Google (AS15169) victim
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The family template (
tools/replays/gen_family_audit.py): UPSTREAM-1 (AS 64500, a network that took Cogent's announcement) has a session to COGENT-1 (AS 174, Cogent, originating a Google /24) with no inbound prefix filter, a prefix limit, no AS-path filter, and a second session with every safeguard, so that each finding is one row. - Nothing public says how the /24 came to be originated. What is auditable is the session that accepted it: modelled with a prefix limit and no inbound prefix filter.
- No change is replayed: the announcing network's change is not ours to gate. The audit runs on the accepting router's healthy configuration.
- Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
UPSTREAM-1's own router, no change: no inbound filter
now: fail (finding)An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.route_map_in_attached. Pinned status: fail, a finding whose evidence contains the text below.
Evidence pinned (the evidence contains each line)
- 1 row(s)
- UPSTREAM-1
- 174
First run, before the fixes this batch prompted
first: fail (finding)1 row(s) returned (expected 0); first: s.device=UPSTREAM-1 s.peer=172.16.0.2 s.remote_as=174
Pinned in tests/gauntlet/replays-audit.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
Inbound max prefix length and AS-path filter on BgpSession
prefix_list_max_length and a new ebgp_inbound_as_path_filter audit stop being stubs: a /32 for somebody else's address wins by longest match wherever it is accepted; a prefix filter generated from a forged registry object says nothing about whose path.
Found by: Accepting-network family
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as partly in scope: A proposed change that adds a network statement or static for address space not in the operator's own allocation list is flaggable, but nothing public says that is what happened.
Imported from docs/replays/cogent-google-2005/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.