HomeIncident library
Incident library

INCIDENT REPLAY · 2012-11-06

Moratel leak of Google routes via PCCW

At about 02:24 UTC on 6 November 2012 (evening of 5 November US Pacific time), Moratel announced paths to Google prefixes to its upstream PCCW, which accepted and propagated them. Google services were unreachable for about 27 minutes for an estimated 3 to 5 percent of internet users, most heavily in Hong Kong. A Cloudflare engineer contacted Moratel and the routes were withdrawn minutes later.

Route leakBGP leak / hijackCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Moratel, Indonesia (AS23947); upstream PCCW (AS3491); victim Google (AS15169)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • At about 02:24 UTC on 6 November 2012 (evening of 5 November US Pacific time), Moratel announced paths to Google prefixes to its upstream PCCW, which accepted and propagated them. Google services were unreachable for about 27 minutes for an estimated 3 to 5 percent of internet users, most heavily in Hong Kong. A Cloudflare engineer contacted Moratel and the routes were withdrawn minutes later.
  • Who: Moratel, Indonesia (AS23947); upstream PCCW (AS3491); victim Google (AS15169)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_leak.py): MORATEL-1 (AS 23947) learns the victim's routes over a direct peering with the victim, and PCCW-1 (AS 3491) prefers what MORATEL-1 sends it and filters nothing on that session. A transit carries the victim's routes to the same network the ordinary way, and a third party sits behind it.
  • The edit is assumed. Before the change MORATEL-1's export toward PCCW-1 is a prefix-list permitting its own block; after it, the list is no longer attached. No operator in this family has published what was changed.
  • Cisco IOS syntax, and stand-in address space except where a prefix is on the record.
  • The accepting network's other sessions are filtered and limited, so that the audit's finding is about this session and not the template.
  • Moratel is taken to have learned Google's routes over a direct peering.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

every network in one snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • MORATEL-1 would announce to 172.16.3.1 (AS 3491) routes it learns from other networks
  • path change:
  • now through AS 23947

First run, before the fixes this batch prompted

first: fail
  • forwarding-semantics gate: 1 device(s) changed configuration that reroutes or rewrites traffic without changing any route or ACL — MORATEL-1: BGP peerings and their policy attachments pass --allow-flow-changes to accept it
  • path change: 1 destination(s) still delivered would be reached through a network their traffic does not cross today — 198.18.0.0/24 on GOOGLE-1 (198.18.0.1) from PCCW-1, X-1 now through AS 23947. No flow is lost, which is what a route leak looks like from inside the model. Pass --allow-path-changes if moving this traffic is intended

Pinned in tests/gauntlet/replays-leak.yaml.

MORATEL-1's own router

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • MORATEL-1 would announce to 172.16.3.1 (AS 3491) routes it learns from other networks

First run, before the fixes this batch prompted

first: pass

Pinned in tests/gauntlet/replays-leak.yaml.

PCCW-1's own router, no change: the audit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • PCCW-1
  • 23947

First run, before the fixes this batch prompted

first: pass (no finding)

Pinned in tests/gauntlet/replays-leak.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: RFC 7908 Type 4 (peer routes to a transit provider): an export policy toward a provider that does not restrict to own and customer routes is detectable, as is PCCW's unfiltered customer session.

Imported from docs/replays/moratel-2012/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.