HomeIncident library
Incident library

INCIDENT REPLAY · 2011-01-14

Indosat mis-originates about 2,800 prefixes

From 12:19 to 12:57 UTC AS4761, which normally originated about 100 prefixes, originated about 2,800 belonging to 824 other ASes, including 8.8.8.0/24, Akamai, Amazon and US government space. Some routes reached up to 50 dispersed vantage points.

Mis-originationHijack originationCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Indosat (AS4761) caused; accepted by AS9505 (Taiwan Internet Gateway), AS6762 (Telecom Italia Sparkle), AS3491; 824 origin ASes affected

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • From 12:19 to 12:57 UTC AS4761, which normally originated about 100 prefixes, originated about 2,800 belonging to 824 other ASes, including 8.8.8.0/24, Akamai, Amazon and US government space. Some routes reached up to 50 dispersed vantage points.
  • Who: Indosat (AS4761) caused; accepted by AS9505 (Taiwan Internet Gateway), AS6762 (Telecom Italia Sparkle), AS3491; 824 origin ASes affected

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_misorigination.py): INDOSAT-1 (AS 4761), an upstream TIG-1 (AS 9505) that prefers its customer's routes and filters nothing on that session, the victim, and a third party behind the upstream. INDOSAT-1 null-routes thirty stand-in /20s that a single router originates on behalf of every network whose routes were claimed.
  • Cisco IOS syntax. No operator here has published configuration, and which vendors they ran is not on the record.
  • The upstream's other sessions are filtered and limited, so that the audit's finding is about this session and not the template.
  • INDOSAT-1 exports with the ordinary customer policy, an AS-path filter that passes only what it originates (^$). A mis-originated prefix is locally originated, so it passes.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

every network in one snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • INDOSAT-1 would start originating 30 prefixes that this network learns from AS 9505, originated by AS 64510
  • connected subnet(s) reachable before are not reachable after

First run, before the fixes this batch prompted

first: fail
  • reachability regression: 1 connected subnet(s) reachable before are not reachable after — 198.18.0.0/24 on INET-1 (198.18.0.1) from INDOSAT-1, TIG-1, X-1 [icmp echo, tcp/179 bgp, tcp/22 ssh, tcp/80 http, udp/53 dns]
  • external advertisements: INDOSAT-1 would start originating 198.18.0.0/20, which is inside 198.18.0.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.112.0/20, which is inside 198.18.112.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.128.0/20, which is inside 198.18.128.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.144.0/20, which is inside 198.18.144.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.16.0/20, which is inside 198.18.16.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.160.0/20, which is inside 198.18.160.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.176.0/20, which is inside 198.18.176.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.192.0/20, which is inside 198.18.192.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.208.0/20, which is inside 198.18.208.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.224.0/20, which is inside 198.18.224.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.240.0/20, which is inside 198.18.240.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.32.0/20, which is inside 198.18.32.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.48.0/20, which is inside 198.18.48.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.64.0/20, which is inside 198.18.64.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.80.0/20, which is inside 198.18.80.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.18.96.0/20, which is inside 198.18.96.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.0.0/20, which is inside 198.19.0.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.112.0/20, which is inside 198.19.112.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.128.0/20, which is inside 198.19.128.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.144.0/20, which is inside 198.19.144.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.16.0/20, which is inside 198.19.16.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.160.0/20, which is inside 198.19.160.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.176.0/20, which is inside 198.19.176.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.192.0/20, which is inside 198.19.192.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.208.0/20, which is inside 198.19.208.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.32.0/20, which is inside 198.19.32.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.48.0/20, which is inside 198.19.48.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.64.0/20, which is inside 198.19.64.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.80.0/20, which is inside 198.19.80.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce
  • external advertisements: INDOSAT-1 would start originating 198.19.96.0/20, which is inside 198.19.96.0/20 that it learns from AS 9505, originated by AS 64510. A more-specific of address space you receive from a peer is not yours to announce

Pinned in tests/gauntlet/replays-misorigination.yaml.

INDOSAT-1's own router

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • INDOSAT-1 would announce 31 prefixes to
  • up from 1

First run, before the fixes this batch prompted

first: unknown

Pinned in tests/gauntlet/replays-misorigination.yaml.

TIG-1's own router, no change: the audit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • TIG-1
  • 4761

First run, before the fixes this batch prompted

first: pass (no finding)

Pinned in tests/gauntlet/replays-misorigination.yaml.

every network in one snapshot (potential microloop)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail.

This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

  • INDOSAT-1 -> TIG-1 -> INDOSAT-1, if TIG-1 has its new next hop while INDOSAT-1 still has its old one

Pinned in tests/gauntlet/replays-transients.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

ROA maximum length

Roa::covers_prefix refused to cover a more-specific beyond maxLength, so such a prefix came back NotFound instead of Invalid (RFC 6811).

Found by: Mis-origination family

--roas <FILE> origin-validates new originations

From the originator's own seat a block that escapes is one new prefix, and only a caveat. With a validator's export it is decidable: "the ROA for 104.244.42.0/24 authorises AS 13414".

Found by: Mis-origination family

An announcement that multiplies

The announcement to an external peer goes from a few prefixes to many in one change.

Found by: Mis-origination family, Level 3

Outbound policy runs before the local AS is prepended

The simulator prepended first, so an outbound filter-list with ^$ matched nothing the router originated.

Found by: Mis-origination family

Many foreign originations are one reason

A mass mis-origination of hundreds of prefixes was hundreds of reasons.

Found by: Mis-origination family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A recurring origination leak from one AS through the same transits is the textbook case for a pre-push originated-prefix diff plus an audit of the transit's customer filters.

Not fixed: A single-prefix hijack from the hijacker's own seat, with no ROA and nothing declared

One new origination is what a legitimate new prefix looks like. pass with the caveat naming it; --roas or --boundary decides it.

Recorded against: Mis-origination family (17 of 28)

Not fixed: A forged-origin hijack

Origin validation passes it by construction. ASPA is not modelled.

Recorded against: Mis-origination family

Imported from docs/replays/indosat-2011/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.