HomeIncident library
Incident library

INCIDENT REPLAY · 2023-05-25

Angola Cables leak slows Australia to US-East traffic

From about 10:30 UTC for roughly three hours, Angola Cables leaked hundreds of routes, many IPv6, through an exchange route server with short AS paths, so they were preferred. Australian access to AWS and Akamai on the US east coast was badly degraded.

Route leakLeak readvertisementCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Angola Cables (AS37468) caused; Akamai (AS20940) and Amazon (AS16509) routes affected; Australian users hurt

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • From about 10:30 UTC for roughly three hours, Angola Cables leaked hundreds of routes, many IPv6, through an exchange route server with short AS paths, so they were preferred. Australian access to AWS and Akamai on the US east coast was badly degraded.
  • Who: Angola Cables (AS37468) caused; Akamai (AS20940) and Amazon (AS16509) routes affected; Australian users hurt

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_leak.py): ANGOLA-1 (AS 37468) learns the victim's routes over a direct peering with the victim, and IX-RS-1 (AS 64500) prefers what ANGOLA-1 sends it and filters nothing on that session. A transit carries the victim's routes to the same network the ordinary way, and a third party sits behind it.
  • The edit is assumed. Before the change ANGOLA-1's export toward IX-RS-1 is a prefix-list permitting its own block; after it, the list is no longer attached. No operator in this family has published what was changed.
  • Cisco IOS syntax, and stand-in address space except where a prefix is on the record.
  • The accepting network's other sessions are filtered and limited, so that the audit's finding is about this session and not the template.
  • Many of the leaked routes were IPv6. The IPv4 shape is replayed here; the IPv6 variant (tools/replays/gen_family_ipv6.py) makes the same edit in the IPv6 address family, over IPv6 sessions, and fails for the same cause.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

IPv6 variant: the leak in the IPv6 address family, every network in one snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • policy widening: ANGOLA-1 would announce to 2001:db8:16:2::1 (AS 64500) routes it learns from other networks, which its IPv6 export policy stopped before

Pinned in tests/gauntlet/replays-ipv6.yaml.

IPv6 variant: ANGOLA-1's own router

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • policy widening: ANGOLA-1 would announce to 2001:db8:16:2::1 (AS 64500) routes it learns from other networks, which its IPv6 export policy stopped before

Pinned in tests/gauntlet/replays-ipv6.yaml.

control: the IPv4 leak on the same dual-stack estate still fails for its cause

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • ANGOLA-1 would announce to 172.16.2.1 (AS 64500) routes it learns from other networks

Pinned in tests/gauntlet/replays-ipv6.yaml.

every network in one snapshot

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • ANGOLA-1 would announce to 172.16.3.1 (AS 64500) routes it learns from other networks
  • path change:
  • now through AS 37468

First run, before the fixes this batch prompted

first: fail
  • forwarding-semantics gate: 1 device(s) changed configuration that reroutes or rewrites traffic without changing any route or ACL — ANGOLA-1: BGP peerings and their policy attachments pass --allow-flow-changes to accept it
  • path change: 1 destination(s) still delivered would be reached through a network their traffic does not cross today — 198.18.0.0/24 on AKAMAI-1 (198.18.0.1) from IX-RS-1, X-1 now through AS 37468. No flow is lost, which is what a route leak looks like from inside the model. Pass --allow-path-changes if moving this traffic is intended

Pinned in tests/gauntlet/replays-leak.yaml.

ANGOLA-1's own router

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • ANGOLA-1 would announce to 172.16.3.1 (AS 64500) routes it learns from other networks

First run, before the fixes this batch prompted

first: pass

Pinned in tests/gauntlet/replays-leak.yaml.

IX-RS-1's own router, no change: the audit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • IX-RS-1
  • 37468

First run, before the fixes this batch prompted

first: pass (no finding)

Pinned in tests/gauntlet/replays-leak.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

BGP IPv6 unicast modelled

A change to BGP IPv6 was unknown: IPv6 routes flooded over IPv4 sessions with no policy and no communities, and none was installed. One engine now runs both families on IOS, and IPv6 routes reach the IPv6 FIB.

Found by: F-root 2011, Angola Cables and Cloudflare Miami IPv6 variants

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: An automation-generated policy change that widens export toward a route server is the motivating case for gating machine-written changes; origin validation could not help because origins were unchanged.

Imported from docs/replays/angola-cables-2023/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.