INCIDENT REPLAY · 2016-06-20
Telia Carrier router misconfiguration (Europe traffic misrouted)
From about 12:10 UTC Cloudflare detected massive packet loss on Telia Carrier's backbone and shut its Telia ports at 12:30 UTC to route around it. Many European users lost access to services such as Slack and WhatsApp. The Register reported that a Telia engineer misconfigured a key router so that European traffic was sent toward Hong Kong, and that Telia sent an apology note to other network operators; Telia's own detailed report was not published.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: Telia Carrier, AS1299; downstream impact on Cloudflare (AS13335), WhatsApp, Slack and others
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.
Sources
- https://blog.cloudflare.com/a-post-mortem-on-this-mornings-incident/
- https://www.datacenterdynamics.com/en/news/cloudflare-issues-caused-by-human-error-at-telia/
- https://www.datacenterknowledge.com/archives/2016/06/21/global-telia-outage-disrupts-popular-internet-services/
- https://www.theregister.com/2016/06/20/telia_engineer_blamed_massive_net_outage/
- https://www.theregister.com/2016/06/21/cloudflare_apologizes_for_telia_screwing_you_over/
What the public record says
- From about 12:10 UTC Cloudflare detected massive packet loss on Telia Carrier's backbone and shut its Telia ports at 12:30 UTC to route around it. Many European users lost access to services such as Slack and WhatsApp. The Register reported that a Telia engineer misconfigured a key router so that European traffic was sent toward Hong Kong, and that Telia sent an apology note to other network operators; Telia's own detailed report was not published.
- Who: Telia Carrier, AS1299; downstream impact on Cloudflare (AS13335), WhatsApp, Slack and others
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- Telia's report was not published. Press accounts say a misconfigured router sent European traffic toward Hong Kong. We assume a local-preference set on everything heard in Hong Kong, which makes the long way round the best path network-wide.
- Frankfurt, Hong Kong, a European network, an Asian carrier that also reaches Europe, and a customer.
- Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
Hammerhead's own first-versus-now record
- First verdict
- fail, for the cause: the European network's subnet would be reached through AS 64520 (the path-change gate merged with PR #31).
- Verdict now
- the same, and ASIA-NET-1 is also named as a router that carries nothing between others today.
Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.
Frankfurt, Hong Kong, a customer and the networks at each end
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- path change:
- now through AS 64520
Pinned in tests/gauntlet/replays-individual.yaml.
Frankfurt, Hong Kong, a customer and the networks at each end (potential microloop)
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail.
This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.
Caveats pinned (some caveat contains each line; a caveat never moves the verdict)
- TELIA-FRA-1 -> TELIA-HKG-1 -> TELIA-FRA-1, if TELIA-FRA-1 has its new next hop while TELIA-HKG-1 still has its old one
Pinned in tests/gauntlet/replays-transients.yaml.
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as in scope: A router config change that moves a continent's traffic onto a wrong path is a forwarding-path diff a pre-push simulation would surface, although the exact faulty statement was never published.
Imported from docs/replays/telia-2016/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.