Verizon / DQE / Allegheny: optimizer more-specifics leaked through a customer

Who was involved

• 1. Three networks are named, with AS numbers. CF: "An Internet Service Provider in Pennsylvania (AS33154 - DQE Communications) was using a BGP optimizer in their network"; "their customer (AS396531 - Allegheny Technologies Inc)"; "their other transit provider (AS701 - Verizon)".
• 2. The business relationships. CF calls Allegheny DQE's "customer" and Verizon Allegheny's "other transit provider". TE: Allegheny Technologies is "a customer of both DQE and Verizon". REG: "Allegheny is also a customer of Verizon". IGP describes DQE as "a fiber network company that is a subsidiary of Pittsburgh-based Duquesne Light Holdings".
• 3. Verizon's role in the wider Internet. CF: "Verizon (AS701), a major Internet transit provider".

The trigger: more-specifics from a BGP optimizer

• 4. CF: "What exacerbated the problem today was the involvement of a 'BGP Optimizer' product from Noction. This product has a feature that splits up received IP prefixes into smaller, contributing parts (called more-specifics). For example, our own IPv4 route 104.20.0.0/20 was turned into 104.20.0.0/21 and 104.20.8.0/21."
• 5. The split routes were meant to stay inside the network that made them. CF: "By splitting these major IP blocks into smaller parts, a network has a mechanism to steer traffic within their network but that split should never have been announced to the world at large." TE: the optimizer "created routes to Cloudflare services that were only meant to be used within DQE's internal network."
• 6. Why more-specifics win. CF: "Specific routes override more general routes".
• 7. Scale, per REG only: "new internet routes for more than 20,000 IP address prefixes – roughly two per cent of the internet – were wrongly announced". CF gives no count.

How the routes travelled

• 8. CF: "DQE announced these specific routes to their customer (AS396531 - Allegheny Technologies Inc). All of this routing information was then sent to their other transit provider (AS701 - Verizon), who proceeded to tell the entire Internet about these 'better' routes. These routes were supposedly 'better' because they were more granular, more specific."
• 9. IGP, same sequence: DQE "advertised BGP routes to its customer(s), including US-based operator Allegheny Technologies Inc. Allegheny accepted these routes and leaked them to a second transit provider, Verizon, who in turn accepted and propagated them."
• 10. IGP on which party broke which norm: "Routes learned from Peers and Transit Providers should only be announced to Customers"; "Allegheny Technologies could have had route-filtering policies in place to prevent their routers from announcing routes they learned from a ... provider"; "Verizon didn't appear to violate the above rules, but it apparently wasn't implementing another best practice, namely, setting an upper limit on the number of routes accepted from a customer."

What Verizon's session to Allegheny lacked

• 11. No prefix limit. CF: "A BGP session can be configured with a hard limit of prefixes to be received. This means a router can decide to shut down a session if the number of prefixes goes above the threshold. Had Verizon had such a prefix limit in place, this would not have occurred."
• 12. No prefix filtering. CF: "If IRR filtering had been used, none of the networks involved would have accepted the faulty more-specifics. What's quite shocking is that it appears that Verizon didn't implement any of this filtering in their BGP session with Allegheny Technologies". Note the hedge "it appears": this is Cloudflare's inference from the outside, not a statement by Verizon.
• 13. No RPKI origin validation. CF: "The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is." "If Verizon had used RPKI, they would have seen that the advertised routes were not valid, and the routes could have been automatically dropped by the router."
• 14. CF's summary: "The leak should have stopped at Verizon. However, against numerous best practices outlined below, Verizon's lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Linode and Cloudflare."

Effect

• 15. Start time. CF: "Today at 10:30UTC, the Internet had a small heart attack. A small company in Northern Pennsylvania became a preferred path of many Internet routes through Verizon (AS701)".
• 16. Traffic was attracted, then lost to congestion, not to missing routes. CF: "suddenly Verizon, Allegheny, and DQE had to deal with a stampede of Internet users trying to access those services through their network. None of these networks were suitably equipped to deal with this drastic increase in traffic, causing disruption in service." REG: connections "ended up going through DQE and Allegheny, which buckled under the strain, causing traffic to disappear into a black hole."
• 17. Size of the loss. CF: "we observed a loss, at the worst of the incident, of about 15% of our global traffic." REG quotes Cloudflare's CTO: "at its worst, about 10 per cent of our traffic was being directed over to Verizon." Verizon's only statement, to REG: "There was an intermittent disruption in internet service for some [Verizon] FiOS customers earlier this morning. Our engineers resolved the issue around 9am ET."
• 18. Duration. The sources disagree: TE says "for nearly two hours"; REG says "For nearly three hours". CF gives no duration.
• 19. Others affected. CF names "Amazon, Linode and Cloudflare". TE adds "services like Discord, Facebook and Reddit" and "some AWS services".

Resolution

• 20. It was fixed at DQE, not at Verizon. CF: "DQE worked with us on the phone to stop advertising these 'optimized' routes to Allegheny Technologies Inc. ... Once this was done, the Internet stabilized". DQE's statement to REG: "DQE was alerted that a third-party ISP was inadvertently propagating routes from one of our shared customers downstream, impacting Cloudflare's services. We immediately examined the issue and adjusted our routing policy".
• 21. CF on Verizon: "while we tried both e-mail and phone calls to reach out to Verizon, at the time of writing this article (over 8 hours after the incident), we have not heard back from them, nor are we aware of them taking action to resolve the issue."
• 22. REG quotes Job Snijders (NTT): "While it is easy to point at the alleged BGP optimizer as the root cause, I do think we now have observed a cascading catastrophic failure both in process and technologies".

None of the following is on the public record. Each is a choice we made so that the corpus is small, parses, converges and has the same failure mode. If one of them is wrong, the replay is wrong in that respect.

• A1. Vendor and syntax. No source names the router vendor or software of any of the four networks. Every device in the corpus is written in Cisco IOS syntax because that is where this parser records all three inbound controls the incident turns on (prefix-list ... in, route-map ... in, maximum-prefix); see RESULT.md for which of the other 13 parsers do. This says nothing about what anyone runs.
• A2. Names, AS numbers and addresses are placeholders. T (AS 64500, routers T-BR-1, T-PE-1, T-PE-2) stands for the large transit provider, C (AS 64510, CUST-1) for the small multihomed customer, R (AS 64505, R-1) for the regional ISP running the optimizer, CDN (AS 64496, CDN-1) for a content network, and X (AS 64511, X-1) for a third party whose traffic we trace. The AS numbers are from the documentation range (RFC 5398). The content prefix is 198.18.0.0/20, from the benchmarking range (RFC 2544), chosen because the documentation ranges hold nothing shorter than a /24 and the incident needs a /20 that splits into two /21s (fact 4). The other blocks are the three documentation /24s. Link and loopback addresses are RFC 1918.
• A3. Topology. Seven routers: the smallest shape we could find with the same failure path. T is three routers in a triangle (OSPF area 0, iBGP full mesh between loopbacks, next-hop-self): one border router peering with the content network, one customer edge facing C, one customer edge facing X. C is one router with two eBGP sessions. R is one router with a customer session to C, a peering session to the content network and a transit session to T's border router. The real networks are vastly larger and nothing about their internal design is public.
• A4. How T and R reach the content network. We gave both a direct settlement-free peering session with it. The record does not say how Verizon or DQE reached Cloudflare in 2019. What the failure mode needs is only that T has a legitimate, shorter path to the /20 that does not go through C, so that the more-specifics are the only reason traffic moves. We also made R a transit customer of T. Whom DQE bought transit from is not in any source we read. Without some path between R and T other than C, the before snapshot already carries T's traffic for R's own prefix through C (we built that first and saw it), and the replay would then start from a network that is already leaking.
• A5. The optimizer's output is modelled as two static routes redistributed into BGP on R-1 (ip route 198.18.0.0 255.255.248.0 172.16.4.2, the same for 198.18.8.0, and redistribute static route-map OPTIMIZER-INJECT). No source we read describes how the optimizer fed its routes to DQE's routers; our understanding, from memory and not re-read, is that such a product injects them over a BGP session of its own, so they would appear in no router's configuration at all (see RESULT.md, 4d). Two consequences of our modelling, both departures from the incident: - the /21s are ORIGINATED by R (AS path 64505 when R sends them to C). No source we read states which origin AS the real leaked more-specifics carried. Our recollection, which we could not re-read in any fetched document, is that an optimizer of this kind copies the AS path of the covering route, so the real ones would still have shown Cloudflare as origin. CF's RPKI argument (fact 13) rests on maxLength alone, "no matter what the path is", and holds either way. In this corpus the /21s would be RPKI-invalid on origin as well as on length; - the split is of exactly one prefix. REG's "more than 20,000" (fact 7) is not reproduced, so nothing about table growth or prefix-count limits being exceeded by a large margin can be read from this corpus.
• A6. R's outbound policy. R sends C everything it has (a full-table customer session, unfiltered) and sends the content network and T only R's own and C's prefixes (prefix-list OWN-AND-CUSTOMERS out). That filter is present in both snapshots and is ours. Without it R would announce the /21s straight back to the network that owns them, which is a different story from fact 5 ("only meant to be used within DQE's internal network") and fact 8 (announced "to their customer").
• A7. C has no outbound filter, in both snapshots. Fact 10 says Allegheny "could have had route-filtering policies in place"; we take that to mean it had none. In before this already leaks in principle: C re-announces R's routes to T and T's to R. In this corpus no best path anywhere uses those re-announcements, because each is one AS hop longer than a path the receiver already has, and C's own best path to the /20 is via T (equal AS-path length, T wins the tie-break), which BGP does not send back to T. We do not know whether Allegheny was leaking DQE's ordinary routes to Verizon before 24 June 2019.
• A8. T's session policy. T-PE-1's session to C has no inbound prefix-list, no inbound route-map and no maximum-prefix, in both snapshots. That is the latent condition of facts 11 and 12. For contrast, and so that a hygiene check can be judged on false positives as well as on the hit, T-PE-2's session to X has prefix-list CUST-X-IN in and maximum-prefix 10, T-BR-1's session to R has prefix-list CUST-R-IN in and maximum-prefix 50, and T-BR-1's peering session has maximum-prefix 500 and no prefix filter. Whether Verizon filtered its other customers is not on the record. T sets no LOCAL_PREF anywhere. A real transit provider prefers customer routes over peer routes; we left that out because it changes nothing about the more-specifics (longest match is decided before BGP attributes are) and it would add policy we would have had to invent.
• A9. The content network drops its own address space inbound. CDN-1 has prefix-list DENY-OWN-SPACE in on both sessions, in both snapshots. Without it, the simulated CDN-1 accepts the leaked /21s of its own /20 back from T, and a packet to an address in the /20 that no interface on CDN-1 owns goes round T, C and R for ever. That loop is an artifact of a one-router content network, not part of the incident, and leaving it in would have let the tool take credit for catching something that did not happen. RESULT.md records what the tool said about that variant, because what it said is wrong in an instructive way.
• A10. The change. after is before with nine lines added to R-1.cfg (seven of configuration, two ! separators) and nothing else touched (captures/00-config-diff.txt). In reality the optimizer had presumably been running for some time and what changed on the day is not public; CF says only that DQE "was using a BGP optimizer". A before/after pair is how this tool is driven, so we made the optimizer's announcements the change.
• A11. The traced flow. X-1, source 192.0.2.1, to 198.18.8.1, an address on a loopback of CDN-1 inside the upper /21. X is ours; the record names no particular victim network behind Verizon.
• A12. Congestion, loss and duration are not modelled at all. The simulator computes converged forwarding state. It has no notion of link capacity, so facts 16 to 18, which are the actual damage, have no counterpart in anything the tool printed. In the model the detoured traffic is delivered.