HomeIncident library
Incident library

INCIDENT REPLAY · 2023-06-02

Unknown-attribute update resets Juniper sessions (COLT and others)

A route carried a path attribute for a not-yet-standardised feature. Routers that did not understand it passed it on untouched; Junos routers that partly understood it treated it as malformed and reset the whole session, repeatedly, cutting off distant networks.

RedundancyAS path / attribute anomalyCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: A small Brazilian network (AS264366) originated the update; COLT (AS8220) and other carriers running Junos hurt

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • A route carried a path attribute for a not-yet-standardised feature. Routers that did not understand it passed it on untouched; Junos routers that partly understood it treated it as malformed and reset the whole session, repeatedly, cutting off distant networks.
  • Who: A small Brazilian network (AS264366) originated the update; COLT (AS8220) and other carriers running Junos hurt

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • COLT (AS 8220) is one of the carriers the record names. We model one Junos edge with an eBGP session to the network that forwarded the attribute (a stand-in AS, 64510) and an iBGP session to a core router, both without bgp-error-tolerance, which is Junos's default.
  • The healthy variant enables bgp-error-tolerance at protocol level. Whether it would have kept those sessions up depends on how that Junos release classified the attribute, which the record says it partly understood; the audit reports the knob, not the outcome.
  • Only Junos is judged. IOS, IOS XR, NX-OS and EOS implement RFC 7606 by default in current releases, and a release's default is not something a configuration shows, so their sessions carry NULL.
  • No change is replayed: nothing in the record says a configuration change started it. The audit runs on the configuration as it stood.
  • Stand-in address space throughout. A reconstruction, not any operator's configuration, and not a claim that the outage would have been prevented.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

COLT-EDGE-1 as built: no bgp-error-tolerance on the transit session

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_error_tolerance_off. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • COLT-EDGE-1
  • 203.0.113.1
  • 64510

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

the healthy variant: bgp-error-tolerance at protocol level

now: pass

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_error_tolerance_off. Pinned status: pass, the check finds nothing.

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: Not a change, but a config audit can report whether tolerant error handling (Junos bgp-error-tolerance, RFC 7606 behaviour) is enabled on eBGP sessions.

Not fixed: RFC 7606 error handling on other vendors

Junos bgp-error-tolerance is parsed and audited; other vendors' behaviour is a release default with no config line, so the audit answers NULL there.

Recorded against: Juniper 2023, OTC 2025

Imported from docs/replays/juniper-attribute-2023/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.