HomeIncident library
Incident library

INCIDENT REPLAY · 1997-04-25

AS7007 incident (MAI Network Services route leak)

At about 11:30 AM Eastern, AS7007's border router received a full view of roughly 23,000 routes from a downstream ISP, de-aggregated them into /24s, stripped the AS path and re-announced them as originated by AS7007. Because the /24s were more specific than the legitimate routes, a large part of Internet traffic was drawn to AS7007 and black-holed. The bad routes lingered in other networks for hours even after the router was disconnected.

Hand-built replaysBGP leak / hijackCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: MAI Network Services / Florida Internet Exchange (AS7007); propagated via Sprint (AS1239, reported) and AS1790

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • At 11:30 EST "our border router, stamped with AS 7007, recieved a full routing view from a downstream ISP (well, a view contacing 23,000 routes anyway)."
  • "the 7007 router then began to de-aggregate the 23K route view and strip the AS path out of it." More than 73,000 routes reached the MAE-East border router of the upstream.
  • MAI's own account of its error: "There was no distibute list imposed on the downstream" and "Not having a filter imposed on the inbound side was our error."
  • The routes were more specific than the real ones and had a one-hop AS path, so other networks preferred them, and traffic for a large part of the internet was drawn to AS 7007. MAI disabled the session at 11:45, rebooted the router at 12:00 and unplugged it at 12:15; the routes persisted in other networks after that.
  • Wikipedia: the deaggregation to /24s was "probably because of a bug in the affected router".

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The mechanism is assumed. The record says the router deaggregated the view and stripped the AS path, and attributes that to a probable bug. It does not say how. This replay models the mechanism the incident is usually taught with, and that produces the same announcement: BGP redistributed into the IGP on one border router, and the IGP's external routes redistributed back into BGP on another, neither through a route-map. Routes that make that round trip come back as locally originated, AS path empty. We do NOT claim that is what MAI's router did, and we do not model the deaggregation at all.
  • Two MAI border routers, because a router does not install its own redistributed externals, so the round trip needs two. A downstream ISP that sends a full view, a transit that prefers what its customers send it, a third party behind that transit, and one router standing in for the rest of the internet with six /20s standing in for the view. AS 7007 and AS 1239 are the real numbers; every other AS number and every address is from the documentation and benchmarking ranges.
  • MAI takes only a default route from its transit, as a small ISP of 1997 would. It matters: if the transit-facing router also heard the view over eBGP, that route would win its routing table over the OSPF external and nothing would be redistributed back, on a real router as in the model.
  • MAI's outbound filter toward the transit is an AS-path list permitting ^$ and ^64505$ (what MAI and its customer originate). That is our guess at a reasonable 1997 edge policy; it matters because a route whose AS path has been stripped matches ^$ and gets through it.
  • Cisco IOS syntax throughout.

tools/replays/gen_as7007_1997.py writes the corpus.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
The before state already held a forwarding loop, because neighbor X filter-list N out was an unparsed line. With that fixed, the change moved nothing in the model: OSPF externals were never fed back into BGP, so the finding was the configuration's shape.
Verdict now
Change: fail, "redistribution loop: MAI-BR-2 re-originates 10 prefix(es) into BGP (AS 7007) that MAI-BR-1 redistributed from BGP into OSPF ... AS 7007 hears them over eBGP (MAI-BR-1 from AS 64505), and MAI-BR-2 announces them as its own with the AS path they arrived with gone. forwarding loop for ... 198.18.0.0/20 through MAI-BR-1, MAI-BR-2", plus lost flows. Audit of MAI's own routers: fail, the unfiltered customer session MAI named as its error.
Fixed?
Yes. BGP and OSPF are iterated to a joint fixpoint (three rounds here), so the re-origination and the loop are simulated and are the finding; bgp-redistribute-ospf-externals is Modeled. The mechanism is the taught one, not a confirmed one, and the deaggregation (a probable router bug) is not claimed.

Quoted from docs/replays/README.md. “First” is the tool before the fixes this replay prompted.

This replay's runs are pinned in Hammerhead's hand-written Rust tests rather than a manifest, so the record above is the only account quoted here.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

IOS neighbor X filter-list N in|out

An unparsed line, so an edge that announces "only what we and our customers originate" by AS path announced everything it learned.

Found by: AS 7007

Mutual redistribution simulated to a fixpoint

BGP into OSPF on one router and OSPF externals into BGP on another was not simulated (one OSPF, BGP, OSPF pass), so the finding was the shape, and a route-map on either leg was assumed to break the loop without being evaluated. BGP and OSPF are now iterated until neither RIB changes; the redistribute route-map and the OSPF tag reach the BGP seed; match internal|external is parsed (a bare IOS redistribute ospf is internal-only); an eBGP route (AD 20) keeps OSPF's copy out. A finding is a prefix re-originated into BGP that the AS hears over eBGP, or a loop; oscillation or the round cap is unknown. Pinned on tests/gauntlet/mutual_redistribution (unsafe fail, tag guard, own-prefixes and control pass). IS-IS in the loop is still not simulated: bgp-redistribute-isis-feedback.

Found by: AS 7007

Layer-2 knobs and filter-list on EOS, NX-OS, Junos, FRR

The EOS/NX-OS spanning-tree helpers dropped every form they did not model without a trace, so preflight passed on EOS a change it fails on IOS.

Found by: Layer-2 family, AS 7007

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: The missing inbound filter / prefix limit on a downstream customer session and the absence of export filtering are config-visible and flaggable, but the de-aggregation and AS-path stripping were blamed on a router software defect that a config model would not predict.

Not fixed: IS-IS, EIGRP and RIP in the mutual-redistribution loop

BGP and OSPF are iterated to a joint fixpoint. IS-IS, EIGRP, RIP and a Junos export policy are lifted into BGP once; a router taking IS-IS into BGP where another puts BGP into IS-IS is reported (bgp-redistribute-isis-feedback). redistribute bgp into OSPF still takes iBGP-learned routes too, which IOS does only with bgp redistribute-internal.

Recorded against: AS 7007

Imported from docs/replays/as7007-1997/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.