HomeIncident library
Incident library

INCIDENT REPLAY · 2024-07-30

Azure Front Door / DDoS: latent network config sent non-European traffic to the European scrubbing system

After a routine DDoS mitigation, a latent network configuration issue caused traffic from outside Europe to be routed to the DDoS protection system inside Europe, congesting it. Azure Front Door, CDN, the Azure portal and Microsoft 365 services saw errors and latency for roughly 2 hours (with a longer tail). Microsoft's fix was to add missing configuration on network devices so an issue in one geography cannot spread.

Required flowRouting policy errorCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Microsoft Azure (tracking ID KTY1-HW8)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • After a routine DDoS mitigation, a latent network configuration issue caused traffic from outside Europe to be routed to the DDoS protection system inside Europe, congesting it. Azure Front Door, CDN, the Azure portal and Microsoft 365 services saw errors and latency for roughly 2 hours (with a longer tail). Microsoft's fix was to add missing configuration on network devices so an issue in one geography cannot spread.
  • Who: Microsoft Azure (tracking ID KTY1-HW8)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_policy.py): US-CLIENTS-1 (US clients) reaches ORIGIN-1 (the origin) through US-EDGE-1 (the US edge where the diversion route was scoped wrongly), which filters inbound on its outside interface. The intent file names the flow: non-European clients are not sent through Europe on tcp/443.
  • A latent scoping gap in diversion routing sent non-European traffic through Europe once mitigation was active there. Modelled as a static route on the US edge, added when mitigation is enabled, that sends the origin's prefix to the European scrubbing router; the intent is a path constraint that US clients' traffic must not traverse it. The intended diversion covers only European clients' source range, which a destination-only static cannot express, so the intended snapshot leaves the US edge alone.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

the change as intended

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --policy @azure_front_door_2024_07/policy.yaml --allow-flow-changes returned pass.

First run, before the fixes this batch prompted

first: pass

Pinned in tests/gauntlet/replays-policy.yaml.

the change as pushed

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --policy @azure_front_door_2024_07/policy.yaml --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • policy gate: high/critical failure(s) on the 'after' network: non-European clients are not sent through Europe
  • must_not_traverse violated
  • path change: EU-SCRUB-1 carries no traffic between other devices today and would carry it

First run, before the fixes this batch prompted

first: fail
  • policy gate: high/critical failure(s) on the 'after' network: non-European clients are not sent through Europe

Pinned in tests/gauntlet/replays-policy.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

A flow rule names its protocol and port

Every reachability rule probed TCP/80, so an intent about SIP could not tell the push from the intended change: 7 of 11 intended changes failed on main and the one that dropped only the named port passed. protocol: and dst_port: on the rule; the policy gate's reason now carries the violation, not only the policy's name.

Found by: Required-flow family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The defect was missing config on network devices (a latent scoping gap in diversion routing) that a what-if of 'mitigation active in Europe' could expose, but it fired only under attack conditions and the PIR does not describe the device config in replayable detail.

Imported from docs/replays/azure-front-door-2024-07/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.