HomeIncident library
Incident library

INCIDENT REPLAY · 2018-07-06 (also 07-10, 07-11 and later in July)

BGP/DNS hijacks against US payment processors

On several dates the prefixes holding payment processors' authoritative DNS servers were hijacked, and forged answers with very long TTLs were served so resolvers kept the bad records after the routes vanished. The last Vantiv hijack lasted nearly three hours.

Accepting networkMalicious deliberateCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Announcements via an Indonesian ISP and a Malaysian ISP (Oracle names them; not verified here); DNS servers of Datawire, Vantiv/Worldpay and Mercury Payment Systems targeted

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • On several dates the prefixes holding payment processors' authoritative DNS servers were hijacked, and forged answers with very long TTLs were served so resolvers kept the bad records after the routes vanished. The last Vantiv hijack lasted nearly three hours.
  • Who: Announcements via an Indonesian ISP and a Malaysian ISP (Oracle names them; not verified here); DNS servers of Datawire, Vantiv/Worldpay and Mercury Payment Systems targeted

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_audit.py): CARRIER-1 (AS 64500, a network that carried the hijacked routes) has a session to HIJACKER-1 (AS 64510, the announcing network) with no inbound prefix filter, no prefix limit, no AS-path filter, and a second session with every safeguard, so that each finding is one row.
  • Only the missing customer and peer filters at the networks that carried the routes are auditable; modelled as one such session.
  • No change is replayed: the announcing network's change is not ours to gate. The audit runs on the accepting router's healthy configuration.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

CARRIER-1's own router, no change: no inbound filter and no prefix limit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • CARRIER-1
  • 64510

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=CARRIER-1 s.peer=172.16.0.2 s.remote_as=64510

Pinned in tests/gauntlet/replays-audit.yaml.

CARRIER-1's own router, no change: no inbound filter

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.route_map_in_attached. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • CARRIER-1
  • 64510

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=CARRIER-1 s.peer=172.16.0.2 s.remote_as=64510

Pinned in tests/gauntlet/replays-audit.yaml.

CARRIER-1's own router, no change: no prefix limit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.max_prefix_configured. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • CARRIER-1
  • 64510

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=CARRIER-1 s.peer=172.16.0.2 s.remote_as=64510

Pinned in tests/gauntlet/replays-audit.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Inbound max prefix length and AS-path filter on BgpSession

prefix_list_max_length and a new ebgp_inbound_as_path_filter audit stop being stubs: a /32 for somebody else's address wins by longest match wherever it is accepted; a prefix filter generated from a forged registry object says nothing about whose path.

Found by: Accepting-network family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: Only the missing customer and peer filters at the networks that carried the routes are auditable.

Imported from docs/replays/payment-processors-2018/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.