INCIDENT REPLAY · 2010-08-22 (BGPmon post dated 2010-08-23 says 'this Sunday')
Evolva Telecom originates Google Public DNS 8.8.8.0/24
AS30890 originated 8.8.8.0/24 for about seven minutes. BGPmon saw it from 14 RIS peers in four countries.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Evolva Telecom, Romania (AS30890) caused; Google Public DNS prefix 8.8.8.0/24 affected
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.
Sources
What the public record says
- AS30890 originated 8.8.8.0/24 for about seven minutes. BGPmon saw it from 14 RIS peers in four countries.
- Who: Evolva Telecom, Romania (AS30890) caused; Google Public DNS prefix 8.8.8.0/24 affected
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The family template (
tools/replays/gen_family_audit.py): UPSTREAM-1 (AS 64500, Evolva's upstream) has a session to EVOLVA-1 (AS 42575, Evolva, originating Google Public DNS) with no inbound prefix filter, a prefix limit, no AS-path filter, and a second session with every safeguard, so that each finding is one row. - The upstream's missing customer prefix filter is the auditable half; the cause on Evolva's side is not documented.
- No change is replayed: the announcing network's change is not ours to gate. The audit runs on the accepting router's healthy configuration.
- Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
UPSTREAM-1's own router, no change: no inbound filter
now: fail (finding)An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.route_map_in_attached. Pinned status: fail, a finding whose evidence contains the text below.
Evidence pinned (the evidence contains each line)
- 1 row(s)
- UPSTREAM-1
- 42575
First run, before the fixes this batch prompted
first: fail (finding)1 row(s) returned (expected 0); first: s.device=UPSTREAM-1 s.peer=172.16.0.2 s.remote_as=42575
Pinned in tests/gauntlet/replays-audit.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
Inbound max prefix length and AS-path filter on BgpSession
prefix_list_max_length and a new ebgp_inbound_as_path_filter audit stop being stubs: a /32 for somebody else's address wins by longest match wherever it is accepted; a prefix filter generated from a forged registry object says nothing about whose path.
Found by: Accepting-network family
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as partly in scope: Same origin-versus-inventory check as any mis-origination, plus the upstream's missing customer prefix filter; cause not documented.
Imported from docs/replays/evolva-2010/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.