INCIDENT REPLAY · 2009-02-16
SuproNet 252-times AS-path prepend
SuproNet announced a single prefix with its AS number prepended 252 times, producing an AS path of about 256 entries. Cisco IOS routers that received the path and applied their own prepending mishandled the over-long attribute, sent malformed updates, and had BGP sessions reset repeatedly. Global update rates spiked (Renesys measured a peak above 100,000 updates per second, per search-result summaries) and instability lasted until the route was withdrawn.
A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.
What happened
Who: SuproNet, Czech Republic (AS47868); affected: networks worldwide running Cisco IOS with bug CSCsx73770
The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.
Sources
- https://www.bgpmon.net/long-as-paths-causing-commotion/
- https://www.sophos.com/en-us/blog/general-20831
- https://blog.ipspace.net/2009/02/oversized-as-paths-cisco-ios-bug/
- https://www.computerworld.com/article/2531417/how-a-small-czech-company--accidentally--caused-a-global-internet-disruption.html
- https://www.secureworks.com/blog/general-20831
- https://blog.ipspace.net/2009/03/as-path-prepending-technical-details/
What the public record says
- SuproNet (AS 47868), a Czech provider, announced a route whose AS path carried its own AS about 250 times. BGPmon: the AS was prepended 252 times, "resulting in a AS path of 256" by the time collectors saw it.
- BGPmon: "A number of routers that apparently run older software, were not capable to handle these long AS paths and as a result a fair number of BGP sessions started to flap." The trigger on Cisco IOS is tracked as CSCsx73770. Secureworks reports a global update rate peaking at 107,780 per second.
- Secureworks, on the cause: the engineer entered the AS number where the router (a MikroTik) expected a COUNT. The router took the number modulo 256 and prepended that many times. On Cisco IOS the same position takes AS numbers to insert.
What the reconstruction assumed
Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.
Everything below is ours.
- One router, Cisco IOS syntax. The real router was a MikroTik, which this tool did not parse when this replay was written (see the RouterOS variant below), and the bug was in how MikroTik reads one argument. What can be modelled is the configuration's EFFECT: a route-map that prepends the AS 252 times, written out the way IOS would need it. We are testing whether the tool notices the announcement, not whether it would have caught the typo in RouterOS syntax.
- Both upstreams are outside the snapshot.
beforeprepends twice toward the backup upstream, which is ordinary;afterprepends 252 times. 94.125.216.0/21 is the prefix BGP collectors recorded; AS 29113 is the upstream that carried it. The primary upstream's AS is a stand-in.
tools/replays/gen_supronet_2009.py writes the corpus.
What Hammerhead said
Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.
Hammerhead's own first-versus-now record
- First verdict
pass, no reasons, no caveats.- Verdict now
fail: "would announce 94.125.216.0/21 to 203.0.113.5 (AS 29113) with an AS path of 253 hops (was 3)"- Fixed?
- Yes, for the announcement. The typo was made in MikroTik RouterOS syntax. RouterOS is now parsed, and the RouterOS rows fail for the same reason; the line as typed (a prepend count outside RouterOS v6's range) is reported unparsed rather than guessed.
Quoted from docs/replays/README.md. “First” is the tool before the fixes this replay prompted.
RouterOS v6, what the router did: its AS 252 times, as `set-bgp-prepend-path`
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- SUPRO-1 would announce 94.125.216.0/21 to 203.0.113.5 (AS 29113) with an AS path of 253 hops (was 3)
Pinned in tests/gauntlet/replays-routeros.yaml.
RouterOS v6, the line as typed: `set-bgp-prepend=47868` is not guessed
now: failOn the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.
Reason text pinned (some reason contains each line)
- 1 new unparsed line(s)
Text no reason may contain
- AS path of
- policy widening
Pinned in tests/gauntlet/replays-routeros.yaml.
What changed in Hammerhead because of it
From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.
AS-path length of what would be announced
A valid route, nothing withdrawn, no flow lost. Longer than 50 hops and grown in this change is a reason.
Found by: SuproNet
MikroTik RouterOS parsed
Both incidents had to be rebuilt in IOS syntax. A dropped filter rule used to renumber the rest, which read as a policy widening; rules are numbered by their place in the chain.
Found by: SuproNet, MikroTik prepends
Limits
- The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
- A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
- A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
- Why the catalog counts this outage as in scope: A verifier would show the originator's announcement going out with a 250-plus hop AS path, and can audit receivers for a missing maxas-limit, but the worldwide damage came from a vendor software bug in third-party routers.
Not fixed: RouterOS forms that are not read
OSPF v6, the v6 VRF, v7 per-connection redistribution, RFC 9234 roles, the firewall input chain and NAT stay unparsed, so a change touching them is refused, not evaluated. What the 2009 software did with an out-of-range prepend count is not guessed.
Recorded against: SuproNet, MikroTik prepends
Imported from docs/replays/supronet-2009/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.