INCIDENT REPLAY · 2014-08-12

512k day: Verizon de-aggregation overflows router TCAM

Around 07:48 UTC Verizon briefly announced about 15,000 new more-specific /24s (RFC 7908 says about 22,000), pushing the global table past 512,000 routes. Routers whose hardware route memory defaulted to 512K entries fell back to software forwarding or dropped routes, causing scattered slowdowns and outages.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Verizon (AS701, AS705) announced the more-specifics; operators of older Cisco platforms with a 512K IPv4 FIB default hurt (2,587 ASes saw instability per BGPmon)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

Around 07:48 UTC Verizon briefly announced about 15,000 new more-specific /24s (RFC 7908 says about 22,000), pushing the global table past 512,000 routes. Routers whose hardware route memory defaulted to 512K entries fell back to software forwarding or dropped routes, causing scattered slowdowns and outages.
Who: Verizon (AS701, AS705) announced the more-specifics; operators of older Cisco platforms with a 512K IPv4 FIB default hurt (2,587 ASes saw instability per BGPmon)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

Verizon's de-aggregation is modelled at 1:1000: 200 /24s out of a /16 that was announced as one route. The victim's TCAM limit is a declaration (--max-device-routes 150 at that scale); no config states what a router's hardware holds.
Two findings, two seats: from Verizon's own edge the announcement multiplies; on the victim, with its limit declared, the table would exceed it.
Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict: pass from Verizon's seat (new announcements were only a caveat) and, on the victim, unknown without a declared limit.
Verdict now: fail from both seats: the announcement to the peer goes from 1 prefix to 201, and the victim's table would exceed its declared limit.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

Verizon's own edge: the announcement multiplies

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

VZ-EDGE-1 would announce 201 prefixes to
up from 1

Pinned in tests/gauntlet/replays-individual.yaml.

the victim with its hardware limit declared (1:1000)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --max-device-routes 150 --allow-advertisement-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

route-table capacity:
VICTIM-EDGE-1 204 routes (was 4, limit 150)

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
Why the catalog counts this outage as partly in scope: Two findings are in reach: a change that removes aggregation and exposes internal /24s to eBGP (RFC 7908 Type 6), and, on victims, a hardware route-table partition with no headroom against the current table size; the victim-side capacity check needs platform data, not just config.

Imported from docs/replays/512k-day-2014/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.