INCIDENT REPLAY · 2018-11-12

MainOne leak of Google prefixes to China Telecom

Between about 21:10 and 22:35 UTC, MainOne announced routes it had learned from Google (it peers with Google at the IXPN exchange in Lagos) to its provider China Telecom, which accepted and re-announced them; some large transit networks then preferred the leaked path. MANRS counted 212 leaked Google prefixes (ThousandEyes saw 180+, other reports about 150). Traffic to Google Search, G Suite and Google Cloud from many networks was pulled toward China Telecom and dropped, causing a roughly 74-minute disruption.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: MainOne Cable Company, Nigeria (AS37282); propagated by China Telecom (AS4809) and onward to TransTelecom, Russia; victims Google (AS15169) and others

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

On 12 November 2018, from about 21:10 to 22:35 UTC, traffic to Google services from many networks was routed through Russia, China and Nigeria and largely dropped.
BGPmon: MainOne Cable Company (AS 37282) "leaked 212 @google prefixes to China telecom".
MainOne: it was "an error during a planned network upgrade due to a misconfiguration on our BGP filters", corrected within 74 minutes.
MainOne peers with Google at the IXPN exchange in Lagos and buys transit from China Telecom (AS 4809). The routes it learned from Google reached China Telecom, which accepted them and announced them onward; Transtelecom (AS 20485) and others then preferred that path.

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

One MainOne router, one Google edge, China Telecom, Transtelecom, a transit that peers with Google, and a Google user behind Transtelecom. Six /20s stand in for the 212 prefixes. AS 37282, 15169, 4809 and 20485 are the real numbers; the rest, and every address, are from the documentation and benchmarking ranges.
The "misconfiguration on our BGP filters" is modelled as one prefix-list entry: the export list toward China Telecom reads permit 0.0.0.0/0 le 24 where it read permit 203.0.113.0/24. The statement does not say what the error was; this is the smallest edit with that effect.
Transtelecom prefers a peer's route (local-preference 150) to its transit's, which is ordinary practice and is what makes the leaked path win there.
China Telecom's session to MainOne has no inbound filter and no prefix limit, which is what "improperly accepted" implies.
Cisco IOS syntax throughout.

tools/replays/gen_mainone_2018.py writes the corpus.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict: Whole snapshot: fail on the path change. MainOne's own seat: pass.
Verdict now: fail from all three seats: the export policy, traffic "from TTK-1, X-1 now through AS 4809, AS 37282", and China Telecom's audit.
Fixed?: Yes, with no new code. Caught by what Dodo / Telstra forced. The model still DELIVERS over the leaked path; that the real one crossed a national firewall and mostly did not is outside it.

Quoted from docs/replays/README.md. “First” is the tool before the fixes this replay prompted.

every network in one snapshot (harness self-check) (potential microloop)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail.

This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

CT-1 -> TTK-1 -> CT-1, if TTK-1 has its new next hop while CT-1 still has its old one

Pinned in tests/gauntlet/replays-transients.yaml.

every network in one snapshot (harness self-check)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

MAINONE-1 would announce to 172.16.2.1 (AS 4809) routes it learns from other networks

Pinned in tests/gauntlet/replays.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Export policy probed with foreign routes

From the leaker's own seat what it would leak is not in the model. Routes as another network would send them, stopped before and announced after, are a reason. It also names the Optus corpus's root cause.

Found by: Dodo / Telstra, MainOne, Optus

Limits

The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
Why the catalog counts this outage as in scope: The operator says a BGP filter was misconfigured during a planned upgrade; comparing announcements to the China Telecom session before and after the change would show peer-learned Google routes newly exported to a transit provider.

Not fixed: What happens to packets inside a network outside the snapshot

A leaked path that still delivers in the model crossed a national firewall and mostly did not.

Recorded against: MainOne

Imported from docs/replays/mainone-2018/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.