HomeIncident library
Incident library

INCIDENT REPLAY · 2021-10-07

Telia Carrier routing-policy update blackholes aggregates

At 16:00 UTC a routine routing-policy update in the Telia Carrier IP core caused traffic to prefixes inside its aggregates to be blackholed for about 65 minutes. Catchpoint reported BGP events on roughly 500k IPv4 and 32k IPv6 networks (more than half of the IPv4 table) and knock-on impact at Cloudflare, Fastly, Equinix Metal, Firebase and NS1. The config was rolled back at 17:05 UTC and services recovered gradually. Note: the outages-list thread dates the event 7 October; Catchpoint's write-up (published the next day) was summarised to me as 8 October.

Fits no familyRouting policy / filter changeCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Telia Carrier (now Arelion / Twelve99), AS1299

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • At 16:00 UTC a routine routing-policy update in the Telia Carrier IP core caused traffic to prefixes inside its aggregates to be blackholed for about 65 minutes. Catchpoint reported BGP events on roughly 500k IPv4 and 32k IPv6 networks (more than half of the IPv4 table) and knock-on impact at Cloudflare, Fastly, Equinix Metal, Firebase and NS1. The config was rolled back at 17:05 UTC and services recovered gradually. Note: the outages-list thread dates the event 7 October; Catchpoint's write-up (published the next day) was summarised to me as 8 October.
  • Who: Telia Carrier (now Arelion / Twelve99), AS1299

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Telia said a routine routing-policy update dropped traffic to prefixes inside its aggregates and did not publish the policy. We assume the customer import policy stopped matching what customers send (a /20 entry written as a /24), leaving only the aggregates' discard routes.
  • One core, one customer edge, two customers, one peer.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
fail with the customers in the snapshot (their subnets are lost). pass from Telia's own routers, which is the seat the operator has: the customers' routes are not in the model, so dropping them moved nothing.
Verdict now
fail from both seats. From Telia's own routers the import policy is probed with the prefixes it named before the change, and each one it would stop accepting is listed.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the core, a customer edge, two customers and a peer

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • connected subnet(s) reachable before are not reachable after
  • on CUSTOMER-1
  • on CUSTOMER-2

Pinned in tests/gauntlet/replays-individual.yaml.

Telia's own routers, which is what the operator has

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • import policy: TELIA-EDGE-1 would stop accepting 1 prefix(es) from
  • 198.18.16.0/20
  • 198.18.32.0/20

Pinned in tests/gauntlet/replays-individual.yaml.

the core, a customer edge, two customers and a peer (potential microloop)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail.

This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

  • TELIA-CORE-1 -> TELIA-EDGE-1 -> TELIA-CORE-1, if TELIA-EDGE-1 has its new next hop while TELIA-CORE-1 still has its old one

Pinned in tests/gauntlet/replays-transients.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Import policy that stops accepting what it accepted (--allow-import-narrowing)

From the carrier's own seat the customers' routes are not in the model. The BEFORE policy's own permits, and a --boundary declaration's advertises, are sent through both policies. First use the declaration's list has in a verdict.

Found by: Telia 2021

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A routing-policy change on core routers that leaves aggregates announced while the contained more-specifics lose forwarding is exactly the before/after reachability and FIB diff a config-verification gate computes.

Imported from docs/replays/telia-2021/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.