Rogers: a deleted route filter flooded the core

The network

• 1. Rogers' wireless and wireline networks "share a common IP core network". Both kinds of traffic are "processed by the same IP core network". (X-ES, Description of the outage; R22 answer 1(ii) says the same: "one IP network infrastructure, that supports all wireless, wireline and enterprise services".)
• 2. Router roles named in the record: Internet gateway, core and distribution routers. R22: "restarting all the Internet Gateway, Core and Distribution Routers in a controlled manner".
• 3. A distribution router "sits between the access layer that connects end users to the network and the core layer that aggregates all the network traffic". (X-ES glossary.)
• 4. The network is multi-vendor, and the vendors are not public. X-FULL §5.1: "The IP core network implements routers from two different major equipment vendors: [redacted] for the distribution routers and [redacted] for the core routers." R22: "a mixed vendor core network consisting of IP routing equipment from multiple tier one manufacturers".
• 5. The two vendors protect against route overload differently. R22 answer 1(ii): "one IP routing manufacturer uses a design that limits the number of routes that are presented by the Distribution Routers to the core routers. The other IP routing vendor relies on controls at its core routers. The impact of these differences in equipment design and protocols are at the heart of the outage".
• 6. Protocols. X-FULL §6.1: "The Rogers IP core network uses BGP as the exterior gateway protocol to advertise IP routes to other autonomous systems, and OSPF or Intermediate System to Intermediate System protocol as interior gateway protocol (IGP)".

The change

• 7. It was phase six of seven. R22: "the sixth phase of a seven-phase process that had begun weeks earlier. The first five phases had proceeded without incident." Implementation of phase six "started at 2:27AM EDT".
• 8. Three distribution routers were involved. R22 answer 1(ii): "a routing configuration change to three Distribution Routers in our common core network". R22 also: "leading to identifying the three Distribution Routers as the cause".
• 9. The change deleted a filter. R22: "The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers." X-ES: "Rogers staff removed the Access Control List policy filter from the configuration of the distribution routers."
• 10. Why it was deleted. X-ES: "Rogers staff deleted the policy filter that prevented IP route flooding in an effort to clean up the configuration files of the distribution routers."
• 11. What the filter governed: redistribution of BGP routes into OSPF. X-FULL §5.2: "The configuration change led to Border Gateway Protocol (BGP) redistribution of full routing tables into OSPF." X-FULL §6.1: "The standard Rogers configuration for the [redacted] distribution routers allowed the distribution of Internet BGP routes into OSPF. Thus, when the policy filter was deleted from the policy statement, the standard configuration led to the distribution of an excessive number of BGP routes into OSPF. The OSPF link-state advertisements from the distribution router overloaded the core routers with data causing them to crash."
• 12. What the filter was for. X-FULL §6.1: "Rogers stated that the removal of the Access Control List policy filter that distributes the old DNS addresses from BGP into OSPF [redacted]." X-FULL §5.2: the change "enables direct distribution of wireless DNS addresses of the cloud infrastructure into the Open Shortest Path First (OSPF) protocol."
• 13. Risk rating. X-ES: Rogers "had initially assessed the risk of this seven-phased process as 'High.' However, as changes in prior phases were completed successfully, the risk assessment algorithm downgraded the risk level for the sixth phase of the configuration change to 'Low' risk". As a result staff were not "required to conduct additional scrutiny, go through higher levels of approvals, and conduct laboratory testing for this configuration change."
• 14. X-ES: "The change management process, which includes audits of change parameters, failed to flag the erroneous configuration change."

Route counts

• 15. The only numbers on the public record. X-ES footnote 6: "Rogers stated that about 10,000 routes are advertised into the core router when the Access Control List policy filter is present on the distribution router. When this policy filter was removed, a single distribution router released over 900,000 route data into the core routers."

The failure

• 16. Timeline. R22: "At 4:43AM EDT, a specific coding was introduced in our Distribution Routers which triggered the failure of the Rogers IP core network starting at 4:45AM." X-FULL §5.3: "The policy filter was removed from [redacted] at 4:43 EDT on 8 July 2022 (the trigger event) ... Within two minutes, all the Rogers core gateways began failing". X-FULL Annex 1, 4:43: "Rogers committed the first change to the first impacted distribution router, deleting a policy filter". Annex 1, 4:58: "The distribution routers flooded all core routers with routes that exceeded their memory limit and consequently prevented them from processing traffic."
• 17. Mechanism of failure. X-FULL §5.2: the flood "overloaded the core routers and exhausted the central processing unit and memory resources, and ultimately, caused the core routers to crash". R22: equipment "exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down."
• 18. No overload protection. X-ES: "The network failure could have been prevented had the core network routers been configured with an overload limit that specifies the maximum acceptable number of IP routing data the router can support. However, the Rogers core network routers were not configured with such overload protection mechanisms."
• 19. It was not an interoperability bug. X-FULL §6.2: the difference in how the two vendors' routers manage link-state advertisements "should not be misconstrued as an interoperability issue between the two routers".
• 20. Scope and duration. X-ES: the outage "lasted from 4:58 EDT on 8 July 2022 to 7:00 EDT on 9 July 2022 as services were gradually restored. More than 12 million customers lost wireless and wireline services".
• 21. Diagnosis took about 14 hours. X-ES: staff "could not pinpoint the root cause for about 14 hours from the start of the outage", partly because the management network "relied on the Rogers IP core network", and partly because "Rogers had completed multiple configuration changes during the maintenance window", "making it difficult to decide which network change ticket to roll back."

What Rogers changed afterwards

• 22. X-FULL, table of measures: "Implemented a limit for BGP redistribution into OSPF (core router configuration parameter)." and "Implemented a limit for the number of entries in the OSPF database."
• 23. X-ES: Rogers "invested in tools that would help validate router configuration changes", built "a separate physical and logical management network", and announced it would separate the wireless and wireline IP cores.
• 24. X-ES, lessons for other operators, first item: "Implement router overload protection in the IP core and distribution networks." Also: "Conduct lab tests of planned configuration changes and ensure that the lab equipment and test scenarios accurately reflect the production network."

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in seconds. If one of them is wrong, the replay is wrong in that respect.

• A1. Vendor and syntax. Both vendors are redacted (fact 4). Every device in the corpus is written in Cisco IOS syntax because that is where this parser's redistribute bgp ... route-map path is best exercised (100% of lines classified, see POST.md). This says nothing about what Rogers runs. X-FULL's phrase "deleted from the policy statement" is the report's wording and we do not infer a vendor from it.
• A2. How the filter was attached. The record says a filter limited which BGP routes were redistributed into OSPF and that it was deleted (facts 9 to 11). We express that as redistribute bgp 65000 subnets route-map BGP-TO-OSPF with a prefix-list behind the route-map, and we express the change as the same line without route-map BGP-TO-OSPF, with the route-map and prefix-list definitions deleted. On IOS, deleting only the route-map body and leaving the reference would redistribute nothing, which is a different failure; we chose the form whose effect matches fact 11.
• A3. Topology. Three distribution routers is fact 8. Two core routers, one Internet gateway, one upstream transit AS and one services AS are ours. So is every link: each distribution router connects to the gateway, to both cores and to the services router, and the two cores connect to each other. The real core had many more routers; the public figures show none of this.
• A4. BGP layout. The record does not say which BGP sessions the distribution routers held or how they learned the Internet table. We use eBGP from the gateway (AS 64510) to the distribution routers (AS 65000) so that the learned routes are redistributable on real IOS without bgp redistribute-internal, a line this parser does not model. All AS numbers are from the private and documentation ranges.
• A5. What the filter permitted. Fact 12 says it concerned DNS addresses of cloud infrastructure, with the rest redacted. We model the permitted set as a block of service prefixes (172.16.0.0/12 le 24) learned over BGP from a services AS, plus a default route. The default route is ours; it gives the cores a way to reach the Internet in the healthy state.
• A6. Scale. The real figures are about 10,000 routes with the filter and over 900,000 from one distribution router without it (fact 15). The committed corpus is 1:100: 100 service prefixes and 9,000 Internet prefixes, every one a real /24 in the configs and in the simulated tables. The ratio is kept; the absolute size is not. See POST.md for why full scale does not run at this revision.
• A7. Prefixes. The Internet table is 9,000 distinct /24s drawn from routable unicast space by a seeded generator (SplitMix64, seed 20220708), originated by the transit router from Null0 statics. They are placeholders for volume. Whoever holds those blocks in reality has nothing to do with this.
• A8. Capacity figure. No router capacity number is public (fact 17 says only that memory and CPU were exhausted). The 1,000-route limit declared for the core routers in capacity.yaml is ours. At 1:100 it corresponds to a 100,000-route engineering limit, ten times the documented normal load. The verdict does not depend on the exact value: any limit between the healthy count and roughly seventy times it gives the same answer.
• A9. Timing and crash behaviour are not modeled at all. The simulator computes converged routing state. It does not model CPU, memory, LSA flooding rates or a router crashing (facts 16 and 17).