HomeIncident library
Incident library

INCIDENT REPLAY · 2020-09-29

Telstra AS1221 mis-origination of ~500 prefixes (ProtonMail, Quad9 and others)

From 17:50 UTC on 29 September 2020 (early 30 September in Australia) AS1221 originated about 472 to 500 IPv4 prefixes belonging to 266 other networks in about 50 countries, mostly US and UK. The event peaked for about three hours. ProtonMail said about 30 percent of the internet's paths to it pointed at Telstra and that it incurred financial loss; Quad9, Cloudflare and Google Sydney ranges were also named.

Mis-originationBGP leak / hijackCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Telstra (AS1221; propagated largely via Telstra Global AS4637), Australia

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • From 17:50 UTC on 29 September 2020 (early 30 September in Australia) AS1221 originated about 472 to 500 IPv4 prefixes belonging to 266 other networks in about 50 countries, mostly US and UK. The event peaked for about three hours. ProtonMail said about 30 percent of the internet's paths to it pointed at Telstra and that it incurred financial loss; Quad9, Cloudflare and Google Sydney ranges were also named.
  • Who: Telstra (AS1221; propagated largely via Telstra Global AS4637), Australia

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_misorigination.py): TELSTRA-1 (AS 1221), an upstream UP-1 (AS 64500) that prefers its customer's routes and filters nothing on that session, the victim, and a third party behind the upstream. TELSTRA-1 null-routes thirty stand-in /20s that a single router originates on behalf of every network whose routes were claimed.
  • Cisco IOS syntax. No operator here has published configuration, and which vendors they ran is not on the record.
  • The upstream's other sessions are filtered and limited, so that the audit's finding is about this session and not the template.
  • TELSTRA-1 exports with the ordinary customer policy, an AS-path filter that passes only what it originates (^$). A mis-originated prefix is locally originated, so it passes.
  • Telstra said a test prefix set was loaded against a production service. ROAs existed for some of the prefixes in 2020; the run with ROAs shows what origin validation of one's own change says, not that every prefix had one.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

every network in one snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • TELSTRA-1 would start originating 30 prefixes that this network learns from AS 64500, originated by AS 64510
  • connected subnet(s) reachable before are not reachable after

First run, before the fixes this batch prompted

first: fail
  • reachability regression: 1 connected subnet(s) reachable before are not reachable after — 198.18.0.0/24 on INET-1 (198.18.0.1) from TELSTRA-1, UP-1, X-1 [icmp echo, tcp/179 bgp, tcp/22 ssh, tcp/80 http, udp/53 dns]
  • external advertisements: TELSTRA-1 would start originating 198.18.0.0/20, which is inside 198.18.0.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.112.0/20, which is inside 198.18.112.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.128.0/20, which is inside 198.18.128.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.144.0/20, which is inside 198.18.144.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.16.0/20, which is inside 198.18.16.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.160.0/20, which is inside 198.18.160.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.176.0/20, which is inside 198.18.176.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.192.0/20, which is inside 198.18.192.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.208.0/20, which is inside 198.18.208.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.224.0/20, which is inside 198.18.224.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.240.0/20, which is inside 198.18.240.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.32.0/20, which is inside 198.18.32.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.48.0/20, which is inside 198.18.48.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.64.0/20, which is inside 198.18.64.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.80.0/20, which is inside 198.18.80.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.18.96.0/20, which is inside 198.18.96.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.0.0/20, which is inside 198.19.0.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.112.0/20, which is inside 198.19.112.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.128.0/20, which is inside 198.19.128.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.144.0/20, which is inside 198.19.144.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.16.0/20, which is inside 198.19.16.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.160.0/20, which is inside 198.19.160.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.176.0/20, which is inside 198.19.176.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.192.0/20, which is inside 198.19.192.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.208.0/20, which is inside 198.19.208.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.32.0/20, which is inside 198.19.32.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.48.0/20, which is inside 198.19.48.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.64.0/20, which is inside 198.19.64.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.80.0/20, which is inside 198.19.80.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: TELSTRA-1 would start originating 198.19.96.0/20, which is inside 198.19.96.0/20 that it learns from AS 64500, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce

Pinned in tests/gauntlet/replays-misorigination.yaml.

TELSTRA-1's own router

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • TELSTRA-1 would announce 31 prefixes to
  • up from 1

First run, before the fixes this batch prompted

first: unknown

Pinned in tests/gauntlet/replays-misorigination.yaml.

TELSTRA-1's own router, with the validator's ROAs

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --roas @telstra_2020/roas.json returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • RPKI: TELSTRA-1 would start originating 198.18.0.0/20 as AS 1221
  • authorises AS 64510

First run, before the fixes this batch prompted

first: not runnable

error: unexpected argument '--roas' found

Pinned in tests/gauntlet/replays-misorigination.yaml.

UP-1's own router, no change: the audit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • UP-1
  • 1221

First run, before the fixes this batch prompted

first: pass (no finding)

Pinned in tests/gauntlet/replays-misorigination.yaml.

every network in one snapshot (potential microloop)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail.

This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

  • TELSTRA-1 -> UP-1 -> TELSTRA-1, if UP-1 has its new next hop while TELSTRA-1 still has its old one

Pinned in tests/gauntlet/replays-transients.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

ROA maximum length

Roa::covers_prefix refused to cover a more-specific beyond maxLength, so such a prefix came back NotFound instead of Invalid (RFC 6811).

Found by: Mis-origination family

--roas <FILE> origin-validates new originations

From the originator's own seat a block that escapes is one new prefix, and only a caveat. With a validator's export it is decidable: "the ROA for 104.244.42.0/24 authorises AS 13414".

Found by: Mis-origination family

An announcement that multiplies

The announcement to an external peer goes from a few prefixes to many in one change.

Found by: Mis-origination family, Level 3

Outbound policy runs before the local AS is prepended

The simulator prepended first, so an outbound filter-list with ^$ matched nothing the router originated.

Found by: Mis-origination family

Many foreign originations are one reason

A mass mis-origination of hundreds of prefixes was hundreds of reasons.

Found by: Mis-origination family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: Simulating the proposed static-route upload on the edge router would show about 500 foreign prefixes being redistributed and announced to peers and transit, a finding a pre-push gate can fail on.

Not fixed: A single-prefix hijack from the hijacker's own seat, with no ROA and nothing declared

One new origination is what a legitimate new prefix looks like. pass with the caveat naming it; --roas or --boundary decides it.

Recorded against: Mis-origination family (17 of 28)

Not fixed: A forged-origin hijack

Origin validation passes it by construction. ASPA is not modelled.

Recorded against: Mis-origination family

Imported from docs/replays/telstra-2020/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.