Microsoft WAN: an IP change made every WAN router recompute

Scope and timeline

• 1. PIR, What happened?: "Between 07:08 UTC and 12:43 UTC on 25 January 2023, customers experienced issues with network connectivity, manifesting as long network latency and/or timeouts when attempting to connect to resources hosted in Azure regions, as well as other Microsoft services including Microsoft 365 and Power Platform." It "also impacted Azure Government cloud services that were dependent on Azure public cloud." (PRE gave the start as 07:05 UTC.)
• 2. PIR: "While most regions and services had recovered by 09:05 UTC, intermittent packet loss issues caused some customers to continue seeing connectivity issues due to two routers not being able to recover automatically. All issues were fully mitigated by 12:43 UTC."
• 3. Traffic affected, PIR: "to/from the Internet, Inter-Region traffic, Cross-premises traffic via ExpressRoute or VPN/vWAN and US Gov Cloud services using commercial/public cloud services."
• 4. Services named by BC as affected: Microsoft Teams, Exchange Online, Outlook, SharePoint Online, OneDrive for Business, Power BI, Microsoft 365 Admin Center, Microsoft Graph, Microsoft Intune, Microsoft Defender for Cloud Apps and Microsoft Defender for Identity.

The change

• 5. PRE described it as an address change: "As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them."
• 6. PIR is more specific. "At 07:08 UTC a network engineer was performing an operational task to add network capacity to the global Wide Area Network (WAN) in Madrid. The task included steps to modify the IP address for each new router, and integration into the IGP (Interior Gateway Protocol, a protocol used for connecting all the routers within Microsoft's WAN) and BGP (Border Gateway Protocol, a protocol used for distributing Internet routing information into Microsoft's WAN) routing domains."
• 7. The record names the protocols only as "IGP" and "BGP". It does not say which IGP.

The command

• 8. PIR: "This change added a command to purge the IGP database – however, the command operates differently based on router manufacturer. Routers from two of our manufacturers limit execution to the local router, while those from a third manufacturer execute across all IGP joined routers, ordering them all to recompute their IGP topology databases."
• 9. PRE: "The command that caused the issue has different behaviors on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed."
• 10. Three router manufacturers are in the WAN (fact 8, and the repair item in fact 19: "across all three vendors for all WAN router roles"). None is named.
• 11. The command was not on the block list. PIR: "Microsoft has a real-time Authentication, Authorization, and Accounting (AAA) system that must approve each command run on each router, including a list of blocked commands that have global impact", but "the command's different, global, default action on the router platform being changed was not discovered during the high-impact commands evaluation for this router model and, therefore, had not been added to the block list."

The process

• 12. PIR: the standard operating procedure "follows a 4-step process that involves: [1] testing in our Open Network Emulator (ONE) environment for change validation; [2] testing in the lab environment; [3] a Safe-Fly Review documenting steps 1 and 2, as well as a roll-out and roll-back plans; and [4] Safe-Deployment which allows access to only one device at a time, to limit impact."
• 13. PIR: "In this instance, the SOP was changed prior to the scheduled event, to address issues experienced in previous executions of the SOP. Critically, our process was not followed as the change was not re-tested and did not include proper post-checks per steps 1-4 above."

The failure

• 14. It was not a wrong end state. PIR: "even though the engineer only had access to a single router, it was still connected to the rest of the Microsoft WAN via the IGP protocol. Therefore, the change resulted in two cascading events. First, routers within the Microsoft global network started recomputing IP connectivity throughout the entire internal network. Second, because of the first event, BGP routers started to readvertise and validate prefixes that we receive from the Internet. Due to the scale of the network, it took approximately 1 hour and 40 minutes for the network to restore connectivity to every prefix."
• 15. The network recovered by itself. PIR: "During the time it took for routing to automatically converge, customer impact dynamically changed as the network completed its convergence." PRE: "By 08:10 UTC, the network started to recover automatically."
• 16. It happened twice. PIR: "Issues in the WAN were detected by monitoring and alerts to the on-call engineers were generated within 5 minutes of the command being run. However, the engineer making changes was not informed due to the unqualified changes to the SOP. Due to this, the same operation was performed again on the second Madrid router 33 mins after the first change, thus creating two waves of connectivity issues".
• 17. The tail. PIR: "our automated systems for maintaining the health of the WAN were paused, including the systems for identifying and removing unhealthy devices and the traffic engineering system for optimizing the flow of data across the network. Due to the pause in these systems, some paths in the network were not fully optimized and, therefore, experienced increased packet loss from 09:25 UTC until those systems were manually restarted".
• 18. What it looked like from outside. NW, reporting ThousandEyes: "Multiple Microsoft BGP prefixes were withdrawn completely and then almost immediately readvertised", and "These re-advertisements repeated several times, causing significant route-table instability."

What Microsoft changed afterwards

• 19. PIR names two contributing factors: "A change was made to a standard operating procedure that was not properly revalidated and left the procedure containing an error and without proper pre- and post- checks." and "A standard command that has different behaviors on different router models was issued outside of standard procedure that caused all WAN routers in the IGP domain to recompute reachability." Repair items include: "Audit and block similar commands that can have widespread impact across all three vendors for all WAN router roles", and "Publish real-time visibility of approved-automated and approved-break glass, as well as unqualified device activity".
• 20. None of the repair items is a configuration check. They are a command block list, change visibility, training, and review of procedures.

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in seconds. If one of them is wrong, the replay is wrong in that respect.

• A1. The real incident was not a wrong end state, and this corpus cannot contain it. Facts 8, 14 and 15: an operational command (not a configuration line) made every router in the IGP domain recompute, the network was impaired while it reconverged, and it then recovered with no configuration change. A pair of config snapshots has nowhere to put a command, a router model's interpretation of it, or a recomputation in progress. What we CAN express in config terms is the planned change the command was part of (facts 5 and 6: an IP address change on a WAN router), done right and done wrong. Those are the candidates below. The "done wrong" candidates are our invention. Nothing on the record says Microsoft's address change was itself incorrect.
• A2. The change is a renumbering of an existing router. Fact 6 says the routers were new and being integrated. We model an in-service router because a router that carries no traffic yet has nothing to lose, and the brief for this replay is the address change. Two forms, each with an intended and a botched candidate:

- Pair 1, a core link. The wan-par-1 to wan-mad-1 link moves from 10.0.0.12/31 to 10.0.0.112/31. after_intended/ renumbers both ends. after_botched/ renumbers only wan-mad-1's end, so the two ends are in different subnets and the OSPF adjacency cannot form. - Pair 2, a loopback. wan-mad-1 Loopback0 moves from 10.255.0.5 to 10.255.0.65, and its OSPF and BGP router-ids with it. after_intended_loopback/ also updates every reference on other routers: both route reflectors' neighbor statements, both reflectors' BGP-protection ACL entries, and a static route on wan-mad-2 whose next hop is the loopback. after_botched_loopback/ updates wan-mad-1's own file and nothing else, as if the peer-side steps of the procedure were skipped.

• A3. Vendor and syntax. Three unnamed manufacturers (fact 10). Every device here is Cisco IOS syntax, because that is where this parser classifies the most of what such a router carries. This says nothing about what Microsoft runs, and a single-vendor corpus cannot represent the thing that mattered on the day, which was a difference between vendors.
• A4. The IGP is OSPF because the model could not run iBGP over IS-IS. The record says "IGP" (fact 7). We first built the WAN on IS-IS level 2. The simulator then refused all 13 iBGP sessions, because its BGP stage checked peer reachability against OSPF results only (RESULT.md, section 6.1). That snapshot is kept as probes/before_isis/. The committed corpus uses OSPF area 0 so that the replay ran at all. This is a limit of the tool that changed the corpus, so we say so here rather than leave OSPF looking like a free choice. The limit (G5) was fixed in PR #86 (60df365): probes/before_isis now establishes its 13 iBGP sessions. The corpus was not rebuilt on IS-IS, so every recorded result still describes the OSPF WAN.
• A5. Topology. Madrid and two Madrid routers are on the record (facts 6 and 16). Everything else is ours: eight WAN routers in one AS (wan-lon-1, wan-ams-1, wan-fra-1, wan-par-1, wan-mad-1, wan-mad-2, wan-dub-1, wan-nyc-1), eleven /31 core links with the metrics in gen_corpus.py, and four regional sites (site-fra-1, site-dub-1, site-mad-1, site-nyc-1), each its own AS with eBGP to one or two WAN routers. site-mad-1 is dual-homed to both Madrid routers. The real WAN has thousands of routers ("Due to the scale of the network", fact 14); the public record shows none of its shape.
• A6. BGP layout. iBGP between loopbacks with two route reflectors (wan-lon-1, wan-ams-1); every other WAN router is a client of both and sets next-hop-self. The reflectors carry no site and do not set next-hop-self: with it, the model rewrites the next hop of reflected routes and reports a forwarding loop that IOS would not have (RESULT.md, section 6.2; probes/rr_next_hop_self/). All AS numbers are private-use; 65010 is the WAN. Microsoft's real AS is not used.
• A7. Addresses. 10.255.0.0/24 loopbacks, 10.0.0.0/24 core links, 172.20.0.0/24 site links, 10.101.0.0/16 to 10.104.0.0/16 site prefixes, 10.250.5.0/24 a Madrid telemetry LAN. All RFC 1918. The telemetry LAN, the static route on wan-mad-2 that reaches it through wan-mad-1's loopback, and the reflectors' BGP-PEERS-IN ACL exist so that the corpus contains the three kinds of reference a loopback renumbering has to chase (a BGP neighbor statement, a static next hop, an ACL host entry). They are plausible, and they are ours.
• A8. Lines left out so that parse coverage is 100%. A production config would carry these, and this parser does not classify them on IOS: isis network point-to-point, ip ospf network point-to-point, ip ospf hello-interval, ip ospf dead-interval, timers throttle spf, max-metric router-lsa on-startup, set-overload-bit on-startup, spf-interval, log-adjacency-changes (under OSPF), bgp cluster-id, no bgp default ipv4-unicast, address-family ipv4 (accepted only when spelled address-family ipv4 unicast) with neighbor ... activate under it, mtu, mpls ip, load-interval, no ip redirects, no ip proxy-arp, and the management plane (version, service ..., ip cef, snmp-server, logging host, ntp server, line vty, access-class). We left them out rather than carry unparsed lines. Note which ones they are: every knob that governs how fast and how gracefully a router reconverges is in that list.
• A9. Timing, flooding and recomputation are not modeled at all. The simulator computes converged routing state. It has no clock, no LSA or LSP flooding, no SPF scheduling, no BGP update churn, and no notion of a router being unable to forward while it recomputes (facts 5, 14, 15, 18).
• A10. The command and the vendor, for `COMMANDS.md`. Microsoft names neither (facts 8 and 10); it says only that the command purges the IGP database, is local on two vendors and orders every IGP-joined router to recompute on the third. For the operational-command rows we ASSUME the command was Junos clear ospf database purge on a Juniper wan-mad-1. It is chosen because it is one of the two IGP purges we found a vendor documenting as flooded to the other routers, and the only one for OSPF, which is the IGP this corpus runs (A4): JUNOS says "All link-state advertisements are set to MAXAGE and are flooded. The database is repopulated when the originators of the link-state advertisements receive the MAXAGE link-state advertisements and reissue them", and warns that it "causes significant network disruption". How far a flooded LSA travels is the protocol's to say: RFC 2328 12.1.3 floods every type but AS-external LSAs "throughout a single area only", so the check reads the purge as reaching the routers in wan-mad-1's areas, which in this one-area corpus is every WAN router. (The other is Arista EOS clear isis database, which EOS says "sends purge LSPs throughout the network to clear LSPs from all devices", and RFC 6232 floods "throughout an area"; using it would have meant an IS-IS corpus.) The purge option is on the Junos 12.x and 13.2 pages and no longer on the current one. Nothing on the record says Microsoft runs Juniper, or that this is the command. tests/gauntlet/microsoft_wan_2023/ multivendor/ is the WAN of before/ and after_intended/ with wan-mad-1 written in Junos syntax; to parse it, the site link is a passive OSPF interface instead of the reflector sessions setting next-hop-self (the Junos parser reads no then next-hop self), and the site ACL is a firewall filter. Every other router is unchanged.