HomeIncident library
Incident library

INCIDENT REPLAY · 2012-02-23

Dodo / Telstra: a customer re-announces the full table

A customer's export filter stopped filtering; it re-announced the full table to Telstra, which prefers customer routes and passed it on until its upstreams' prefix limits tore the sessions down.

Hand-built replaysNot classified (hand-built)Catalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

The summary at the top is Hammerhead's own one-line description of the replay; the sources below are the record. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • "all of Telstra's prefixes suddenly disappeared at 2:40 UTC and came back approximately 30 minutes later".
  • "Dodo a Telstra customer, re-announced all Internet routes to Telstra". Dodo is AS 38285; Telstra is AS 1221.
  • Telstra "because it prefers customer routes now thinks the best way to the Internet is through Dodo".
  • "Telstra announced a full Internet routing table to its upstream providers, triggering the max prefix limit causing the BGP sessions to go down and as a result isolating Telstra and its customers from the rest of the Internet".
  • APNIC Labs describes the general shape: a small ISP takes a full table from one provider and relays it to another because of incorrect filtering, and the second provider has no filter of its own on that customer.

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours. Neither operator has published configuration.

  • One Dodo router with two providers, Telstra and an unnamed "ISP A" (AS 64496 here). A two-router Telstra (customer edge, international border), one upstream of Telstra (AS 64500) with maximum-prefix 6 on its Telstra session, and one router standing in for the rest of the internet with eight /20s standing in for the full table. AS 38285 and AS 1221 are the real numbers; every other number and address is from the documentation and benchmarking ranges.
  • The change is one line on Dodo's router: the outbound prefix-list toward Telstra is no longer attached. The record says "incorrect filtering" and no more, so the exact edit is a guess at the simplest one.
  • Telstra's customer session has no inbound prefix filter and no prefix limit, and a route-map that raises local-preference to 200. The preference for customer routes is on the record; the absence of a filter is what the outcome implies.
  • The upstream's limit of 6 against a normal 2 is scaled to the corpus.
  • Cisco IOS syntax throughout.

tools/replays/gen_dodo_telstra_2012.py writes the corpus.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
Whole snapshot: fail for both real reasons on the first run. Telstra's audit: missed the Dodo session. Dodo's own seat: pass once a generic tripwire was waived.
Verdict now
Whole: "INTL-1 from 172.16.3.2: 10 prefixes (was 2, limit 6)" and traffic "now through AS 38285, AS 64496". Audit: exactly T-PE-1 -> 172.16.4.2 (AS 38285). Dodo's seat: fail, "would announce to 172.16.4.1 (AS 1221) routes it learns from other networks, which its export policy stopped before".
Fixed?
Yes, two fixes. The audit took any attached route-map for an inbound filter, including one that only sets local-preference. And from the leaker's own seat nothing it would leak is in the model, so the export policy is now probed.

Quoted from docs/replays/README.md. “First” is the tool before the fixes this replay prompted.

This replay's runs are pinned in Hammerhead's hand-written Rust tests rather than a manifest, so the record above is the only account quoted here.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

A route-map that rejects nothing is not a filter

The hygiene audit missed the one session that mattered because route-map CUSTOMER-IN in (set local-preference, match nothing) counted as an inbound filter.

Found by: Dodo / Telstra

Export policy probed with foreign routes

From the leaker's own seat what it would leak is not in the model. Routes as another network would send them, stopped before and announced after, are a reason. It also names the Optus corpus's root cause.

Found by: Dodo / Telstra, MainOne, Optus

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.

Imported from docs/replays/dodo-telstra-2012/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.