HomeIncident library
Incident library

INCIDENT REPLAY · 2011-10-01

F-root IPv6 route leaks out of Beijing

For about 25 hours the local-scope anycast announcement of F-root's IPv6 prefix from the Beijing node was visible globally, so IPv6 queries to F-root from around the world could be answered from inside China.

IPv6Leak readvertisementCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: ISC F-root Beijing node (AS55439) prefix 2001:500:2f::/48 leaked by an upstream there and picked up by Hurricane Electric (AS6939); IPv6 resolvers worldwide affected

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • For about 25 hours the local-scope anycast announcement of F-root's IPv6 prefix from the Beijing node was visible globally, so IPv6 queries to F-root from around the world could be answered from inside China.
  • Who: ISC F-root Beijing node (AS55439) prefix 2001:500:2f::/48 leaked by an upstream there and picked up by Hurricane Electric (AS6939); IPv6 resolvers worldwide affected.
  • Whether a configuration change caused it: probably. The published account attributes the escape to the upstream in Beijing passing on a route announced for local scope, which is that network's behaviour, not the node's configuration.

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Three routers: FROOT-BJ (AS 55439) announcing 2001:500:2f::/48 for local scope, CN-UP (AS 64600, a stand-in for the Beijing network it peered with) and HE-1 (AS 6939) as the global transit that picked the route up. Sessions over IPv6, active in the IPv6 family only. Nothing about the 2011 configurations is published.
  • The scope is NO_EXPORT, set by FROOT-BJ's outbound route-map toward CN-UP and sent with send-community. Local anycast nodes used NO_EXPORT or an equivalent; which one ISC used in Beijing is not on the record.
  • Both edits are assumed. The upstream's: its inbound policy tags customer routes and REPLACES their communities instead of adding to them, so the scope is gone before CN-UP decides what to pass on. The node's own (a second way to lose the scope, checked from the node's seat): send-community dropped from the session, so the tag never leaves the router. The control tags additively and keeps it.
  • Cisco IOS syntax, documentation address space for the links.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

the node's own router: the scope community stops being sent

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • scope communities: FROOT-BJ would announce 2001:500:2f::/48 to 2001:db8:16::2 (AS 64600) without no-export, which it carried there before

Pinned in tests/gauntlet/replays-ipv6.yaml.

every network in one snapshot: the upstream's inbound policy replaces the scope

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • scope communities: CN-UP would announce 2001:500:2f::/48 to 2001:db8:16:1::2 (AS 6939), which it held with no-export before and did not announce there

Text no reason may contain

  • scope communities: FROOT-BJ

Pinned in tests/gauntlet/replays-ipv6.yaml.

control: the upstream tags customer routes additively and the scope survives

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot returned pass.

Pinned in tests/gauntlet/replays-ipv6.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

BGP IPv6 unicast modelled

A change to BGP IPv6 was unknown: IPv6 routes flooded over IPv4 sessions with no policy and no communities, and none was installed. One engine now runs both families on IOS, and IPv6 routes reach the IPv6 FIB.

Found by: F-root 2011, Angola Cables and Cloudflare Miami IPv6 variants

A route that loses its scope community

Sent without the NO_EXPORT it carried, or announced after being held with one: nothing moves inside the snapshot. Both families, under policy widening.

Found by: F-root 2011

set community no-export in an IOS outbound route-map

Read after export policy, so it stopped the route at its own origin and the peer never heard it. IOS now reads the BGP table's copy; Junos keeps the post-policy reading.

Found by: F-root 2011

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A local anycast node depends on NO_EXPORT or an equivalent scope community being honoured; an export policy that ignores it, or an origin session that does not set it, is directly readable from config.

Not fixed: What BGP IPv6 still lacks

IPv4 routes over an IPv6-transport session are in no table (bgp-ipv4-over-ipv6-transport, blocks); an IPv6 limit crossed on an IPv4-transport session keeps the IPv4 routes where IOS resets the whole session (bgp-ipv6-limit-resets-ipv4-session, reported). Only IOS records BGP IPv6. RPKI, boundary declarations, --allow-withdrawal, foreign originations and prepends, import narrowing, long AS paths and starved sessions still read IPv4 only, and the verdict lists them.

Recorded against: IPv6 family

Imported from docs/replays/f-root-2011/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.