HomeIncident library
Incident library

INCIDENT REPLAY · 2020-08-24

Premier DC (AS42910) repeated route leak

Qrator Radar, as summarised by MANRS, recorded a leak of about 1,403 prefixes that recurred several times over more than four hours.

Accepting networkLeak readvertisementCatalog confidence: low

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Premier DC (AS42910) caused; about 1,403 prefixes, mostly US and Western Asia origins

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account low.

Sources

What the public record says

  • Qrator Radar, as summarised by MANRS, recorded a leak of about 1,403 prefixes that recurred several times over more than four hours.
  • Who: Premier DC (AS42910) caused; about 1,403 prefixes, mostly US and Western Asia origins

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_audit.py): PREMIER-1 (AS 42910, Premier DC, the leaker) has a session to PROVIDER-A-1 (AS 64500, one of its providers) with a prefix-list of the customer's own block, a prefix limit, an inbound AS-path filter, no outbound filter, and a second session with every safeguard, so that each finding is one row.
  • A repeating leak suggests a standing export-policy gap rather than a one-off change. Modelled from the leaker's own seat: a provider session with no outbound filter.
  • No change is replayed: the announcing network's change is not ours to gate. The audit runs on the accepting router's healthy configuration.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

PREMIER-1's own router, no change: no outbound filter

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.route_map_out_attached. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • PREMIER-1
  • 64500

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=PREMIER-1 s.peer=172.16.0.2 s.remote_as=64500

Pinned in tests/gauntlet/replays-audit.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Inbound max prefix length and AS-path filter on BgpSession

prefix_list_max_length and a new ebgp_inbound_as_path_filter audit stop being stubs: a /32 for somebody else's address wins by longest match wherever it is accepted; a prefix filter generated from a forged registry object says nothing about whose path.

Found by: Accepting-network family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: A repeating leak suggests a standing export-policy gap rather than a one-off change; auditable, but detail is thin.

Imported from docs/replays/premier-dc-2020/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.