INCIDENT REPLAY · 2019-06-06
Safe Host leaks 70,000 routes to China Telecom
From 09:44 UTC Safe Host leaked over 70,000 routes learned from peers and providers to China Telecom, which announced them onward. Traffic for large European mobile networks flowed through China Telecom for more than two hours.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Safe Host (AS21217), Switzerland, caused; China Telecom (AS4134) accepted in Frankfurt and propagated; Swisscom (AS3303), KPN (AS1136), Bouygues (AS5410), SFR (AS21502) and others hurt
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.
Sources
- https://blog.apnic.net/2019/06/07/large-european-routing-leak-sends-traffic-through-china-telecom/
- https://blog.apnic.net/2020/07/03/one-year-of-bgp-insecurity/
- https://cert.europa.eu/static/threat-intelligence/TLP-WHITE-CERT-EU-MEMO-190611-1.pdf
- https://manrs.org/2019/06/large-european-routing-leak-sends-traffic-through-china-telecom/
- https://www.theregister.com/2019/06/10/bgp_route_hijack_china_telecom/
What the public record says
- From 09:44 UTC Safe Host leaked over 70,000 routes learned from peers and providers to China Telecom, which announced them onward. Traffic for large European mobile networks flowed through China Telecom for more than two hours.
- Who: Safe Host (AS21217), Switzerland, caused; China Telecom (AS4134) accepted in Frankfurt and propagated; Swisscom (AS3303), KPN (AS1136), Bouygues (AS5410), SFR (AS21502) and others hurt
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- The family template (
tools/replays/gen_family_leak.py): SAFEHOST-1 (AS 21217) learns the victim's routes through UP-1 (AS 64500), and CHINANET-1 (AS 4134) prefers what SAFEHOST-1 sends it and filters nothing on that session. A transit carries the victim's routes to the same network the ordinary way, and a third party sits behind it. - The edit is assumed. Before the change SAFEHOST-1's export toward CHINANET-1 is a prefix-list permitting its own block; after it, the list is no longer attached. No operator in this family has published what was changed.
- Cisco IOS syntax, and stand-in address space except where a prefix is on the record.
- The accepting network's other sessions are filtered and limited, so that the audit's finding is about this session and not the template.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
every network in one snapshot
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- SAFEHOST-1 would announce to 172.16.3.1 (AS 4134) routes it learns from other networks
- path change:
- now through AS
First run, before the fixes this batch prompted
first: fail- forwarding-semantics gate: 1 device(s) changed configuration that reroutes or rewrites traffic without changing any route or ACL — SAFEHOST-1: BGP peerings and their policy attachments pass --allow-flow-changes to accept it
- path change: 1 destination(s) still delivered would be reached through a network their traffic does not cross today — 198.18.0.0/24 on SWISSCOM-1 (198.18.0.1) from CHINANET-1, X-1 now through AS 21217, AS 64500. No flow is lost, which is what a route leak looks like from inside the model. Pass --allow-path-changes if moving this traffic is intended
Pinned in tests/gauntlet/replays-leak.yaml.
SAFEHOST-1's own router
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- SAFEHOST-1 would announce to 172.16.3.1 (AS 4134) routes it learns from other networks
First run, before the fixes this batch prompted
first: passPinned in tests/gauntlet/replays-leak.yaml.
CHINANET-1's own router, no change: the audit
now: fail (finding)An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.
Evidence pinned (the evidence contains each line)
- CHINANET-1
- 21217
First run, before the fixes this batch prompted
first: pass (no finding)Pinned in tests/gauntlet/replays-leak.yaml.
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as in scope: Leaker: export policy toward a peer not limited to own and customer routes. Receiver: a peering session with a data-centre operator accepting 70,000 routes means no max-prefix and no filter.
Imported from docs/replays/safehost-2019/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.