Optus: a partner's table grew tenfold past configured prefix limits

The network

• 1. The outage was in the IP core. SUB para 12: "Connecting the various networks we operate ... is the Internet Protocol (IP) Core network that contains a range of IP routers ... The issue being discussed by this Committee relates to the Optus IP Core network."
• 2. It peers internationally through Singtel exchanges. SUB para 14: "Each separate IP domain uses separate, fully resilient and physically redundant internet gateway routers to connect to Singtel internet exchanges, located in North America and Asia." There are two domains, consumer and enterprise (SUB para 14).
• 3. The layers routes pass through. SUB para 15: "There are certain network elements in each IP network domain called Route Reflectors that act as the control centre for exchange of routing information within and between network domains. These Route Reflectors are connected in a mesh architecture ... IP route information comes into the Optus network through Internet Gateway Routers (IGR), through the Route Reflectors, and onto the Provider Edge (PE) Routers."
• 4. PE routers are dedicated by service. SUB para 16: "dedicated PE Routers for mobile data and voice services (mobile network); dedicated PE Routers for fixed broadband services (fixed network); and dedicated routers for operational support services and the Optus Domain Name System (DNS)."
• 5. About 200 PE routers, more than one vendor. SUB para 16: "In total, Optus has around 200 PE Routers nationally, provided by several global leading vendors".

The failure

• 6. What failed. SUB para 19: "It is now understood that the outage occurred due to approximately 90 PE routers automatically self-isolating in order to protect themselves from an overload of IP routing information. These self-protection limits are default settings provided by the relevant global equipment vendor (Cisco)."
• 7. The trigger was in a partner's network. SUB para 20: "This unexpected overload of IP routing information occurred after a software upgrade at one of the Singtel internet exchanges (known as STiX) in North America ... During the upgrade, the Optus network received changes in routing information from an alternate Singtel peering router. These routing changes were propagated through multiple layers of our IP Core network. As a result, at around 4:05am (AEDT), the pre-set safety limits on a significant number of Optus network routers were exceeded. Although the software upgrade resulted in the change in routing information, it was not the cause of the incident."
• 8. Optus' public statement of 13 November 2023 (quoted in REPORT para 2.21 and ABC-13NOV): "At around 4.05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a routine software upgrade ... These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves."
• 9. The alternate path was configured differently, and a prefix filter was involved. HANSARD, Ms Bayer Rosmarin: "When one of those links was being upgraded and was therefore not available, what was coming through that link needed to be diverted through another link, which happened to be configured differently. It then propagated through our network in a way that triggered these failsafes in each of the different routers." HANSARD (Senator Cadell's questions): "Was the problem around a prefix filter that meant your Cisco router was overloaded?" Mr Kanagaratnam: "The defence on our connectivity to the STiX network—yes, there was an issue with the prefix filter." And: "The defence mechanism that we should have had in place was exactly the filter that we should have had in place. But even if that failed, we wouldn't have expected such a large-scale outage. For us to lose 90 routers in one outage is not something that we contemplated."
• 10. The partner describes an increase in addresses. Singtel media statement, 16 November 2023, as quoted in REPORT para 2.23: "We are aware that Optus experienced a network outage after the upgrade when a significant increase in addresses being propagated through their network triggered preset failsafes. However, the upgrade was not the root cause."
• 11. Each router shut itself down independently, and the limit was not known to the team. HANSARD, Ms Bayer Rosmarin: "the root cause of the issue was that our Cisco routers hit a fail-safe mechanism which meant that each one of them independently shut down. The trigger event that led to that was the upgrade on the SingTel international peering network." And (HANSARD; also quoted in REPORT para 2.30): "we did not have an articulated, clear risk that each one of our 90 routers would independently shut itself down at the same time using a fail-safe on those Cisco routers that our team was not aware of."
• 12. It defeated the redundancy. HANSARD, Mr Kanagaratnam (also quoted in REPORT para 2.27): "there were almost 90 routers across the country that were impacted by the single issue that we had where a preset safety limit on these routers was reached or exceeded. What then happened was they all shut down, and effectively disconnected all these levels of redundancy".
• 13. Cisco's statement, quoted in REPORT para 2.26: "We can confirm that Cisco routers performed as configured and we continue to advise the customer and provide relevant support."
• 14. Optus could not have tested the partner's change. HANSARD, Ms Bayer Rosmarin: "this is an upgrade that's done by a partner, not by us ... We get notified, but we have no ability to influence or test that. Our network has to be designed to cope with the redirection".
• 15. The alarms were about IPv6 prefixes over BGP. SUB para 23(e): "Examining equipment logs and alerts, which showed multiple Border Gateway Protocol (BGP) IPv6 prefixes exceeding threshold alerts." Para 23(f): "Investigating links to potential sources of the high number of IPv6 prefixes including Akamai, to confirm that the route flooding was not ongoing." Para 24: "The IPv6 line of enquiry became our leading hypothesis for network restoration."

Recovery, scope, duration

• 16. The sessions did not come back on their own. SUB para 24: "we identified that resetting routing connectivity addressed the loss of network services. This occurred at 10:21am and we then commenced: (a) Resetting and clearing routing connectivity on network elements which had disconnected themselves from the network. (b) Physically rebooting and reconnecting some network elements". SUB para 21: "more than 100 devices in 14 sites nationwide".
• 17. Timeline. REPORT para 2.3: outage begins 4.05am AEDT; "10.38am: Optus consumer internet services start to come back online"; "4.00pm: services restored for 99 per cent of customers; Optus declares the outage over". That is about 12 hours. (ABC-13NOV says 14 hours; SUB para 29 notes intermittent problems for about 9,000 NBN customers through the night.)
• 18. Scale. REPORT para 2.4: "An estimated 10.2 million Optus subscribers were affected by the outage". Para 2.11: "An estimated 400000 small businesses were affected".
• 19. Enterprise broadband was spared. SUB para 17: "a loss of consumer fixed and mobile services, and some enterprise services. Enterprise broadband services were not impacted as these operate on a different domain."

What Optus changed afterwards

• 20. HANSARD, Mr Kanagaratnam: "We have implemented four key changes on the network. The first one we've done is to make some changes on our international gateway routers that connect to our peering partners. That's to ensure that the routes don't propagate in the same way they did on that morning." The others: more connectivity between the enterprise and consumer networks, more protection for the management network, and a network-wide change freeze.

What a Cisco IOS prefix limit does (vendor documentation, not Optus)

• 21. CISCO-25160: "This feature allows you to control how many prefixes can be received from a neighbor." "By default, this feature allows a router to bring down a peer when the number of received prefixes from that peer exceeds the configured maximum-prefix limit." The example shows the peer in state "Idle (PfxCt)" and says "you must use the clear ip bgp <neighbor-ip> command to restore the peer session." The log line is "%BGP-3-MAXPFXEXCEED: Number of prefixes received from 10.0.0.1 (afi 0): 11 exceeds limit 10".

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in under a second, or something we believe and could not re-read. If one is wrong, the replay is wrong in that respect.

• A1. Vendor and syntax. The vendor of the routers that isolated is public: Cisco (fact 6). The operating system, platform and software release are not. Every device in the corpus is written in Cisco IOS syntax because that is the parser in this tree that records maximum-prefix (crates/hammerhead-parser/src/cisco_ios/mod.rs). It is not a claim that Optus ran IOS. The partner routers are also written in IOS syntax; the record says nothing about Singtel's equipment.
• A2. The limit was a vendor default; in the corpus it is an explicit line. Fact 6 says the limits "are default settings provided by the ... vendor", and fact 11 says the team was not aware of the fail-safe. We REMEMBER, and could not re-read, that Cisco IOS XR applies a per-address-family maximum-prefix to every BGP neighbour with no configuration line at all (from memory: 1,048,576 for IPv4 unicast and 524,288 for IPv6 unicast in current releases, lower in old ones), and tears the session down with a Cease notification when it is exceeded. If that is what happened, the limit that took Optus down appeared in no configuration file. The corpus writes it as neighbor X maximum-prefix 250, the explicit IOS form (fact 21). That is the EASIER case for any config-reading tool, and RESULT.md (gap G5 and section 4d) says what the harder case would need. We do not state which Cisco OS Optus ran or which limit was hit; the record does not say.
• A3. IPv4 stands in for IPv6. The only alarms the record describes are about IPv6 prefixes (fact 15). Hammerhead does not model IPv6 forwarding and its BGP Adj-RIB-In is IPv4-only, so the corpus floods IPv4 /24s. The failure mode (a per-session prefix count crossing a limit) is the same in either family; the address family is not.
• A4. What "configured differently" means. Fact 9 says the alternate link was "configured differently" and that there was "an issue with the prefix filter" on Optus' connectivity to STiX. It does not say what the issue was. We model it as: IGW-1 (primary path) has an inbound prefix-list on its session to STIX-1; IGW-2 (alternate path) has no inbound filter on its session to STIX-2. That difference is present in BOTH snapshots and is harmless until the partner sends more.
• A5. What changed at the partner. The record says a software upgrade made the primary link unavailable and that Optus then received "changes in routing information from an alternate Singtel peering router" (fact 7) and "a significant increase in addresses" (fact 10). It does not say why the alternate sent more. We express it as two edits, both in the partner's files: STIX-1 gets neighbor 198.51.100.2 shutdown (the router under upgrade), and STIX-2 loses its outbound prefix-list, so it advertises its whole table instead of the filtered one. No operator file changes by a byte. An export filter being dropped is our stand-in for "the alternate sent more"; it is not a claim about what Singtel's upgrade did.
• A6. Every number. No prefix count and no limit value is on the public record. 100 partner prefixes normally, 1,000 after, and a limit of 250 are ours, chosen so the healthy state sits well under the limit and the candidate sits well over it. The verdicts do not depend on the values.
• A7. Topology. Ten devices. Two partner routers (primary, alternate) and two Internet gateways follow facts 2 and 7. Two core routers that are also the route reflectors, four PE routers, every link, every address, OSPF as the IGP, and iBGP between loopbacks are ours. The real network had around 200 PE routers (fact 5) and separate route reflectors (fact 3). AS 64496 (documentation) and AS 64512 (private) are placeholders.
• A8. One PE has no limit. About 90 of around 200 PE routers isolated, and the 200 came from "several" vendors (facts 5 and 6). The record does not say the other 110 were a different vendor or why they survived. PE-4 carries no maximum-prefix, so the corpus has one PE that a truthful model should leave standing next to three that it should not.
• A9. PE roles. PE-1 mobile, PE-2 fixed broadband, PE-3 operational support and DNS follow the categories in fact 4. Each has one connected /24 labelled "customer aggregation", advertised into BGP with a network statement. The real services behind a PE are not a /24.
• A10. Prefixes. The partner's normal table is 100 /24s inside 100.64.0.0/10 and the flood is 900 /24s from unicast space outside 100/8, drawn by a seeded generator (SplitMix64, seed 20231108, docs/replays/optus-2023/gen_corpus.py) and originated from Null0 statics. They are placeholders for volume. Whoever holds those blocks in reality has nothing to do with this.
• A11. Timing, teardown and recovery are not modelled. The simulator computes converged steady state. It has no notion of a session going to Idle, of 90 routers tripping within the same minute, of a session staying down until someone clears it (fact 16), or of the management network being unreachable.