HomeIncident library
Incident library

INCIDENT REPLAY · 2025 (Qrator write-up dated 2025-12-15; event dates not given)

Malformed OTC attribute causes session resets

Qrator traced unexplained eBGP session resets to updates whose OTC attribute had an Extended Length flag inconsistent with its length encoding. Implementations differed: some discarded the route, some rewrote the attribute, some reset the session as RFC 4271 originally required.

RedundancyAS path / attribute anomalyCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Unnamed networks; implementations that reset on a malformed RFC 9234 Only-To-Customer attribute

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • Qrator traced unexplained eBGP session resets to updates whose OTC attribute had an Extended Length flag inconsistent with its length encoding. Implementations differed: some discarded the route, some rewrote the attribute, some reset the session as RFC 4271 originally required.
  • Who: Unnamed networks; implementations that reset on a malformed RFC 9234 Only-To-Customer attribute

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The networks are not named. We model one Junos edge (exported in display set form) with two eBGP groups: a customer group with bgp-error-tolerance and a peer group without it, which is the session the finding names.
  • The healthy variant sets bgp-error-tolerance malformed-route-limit 10000 on the peer group as well.
  • This incident rests on one write-up with no event dates.
  • No change is replayed: nothing in the record says a configuration change started it. The audit runs on the configuration as it stood.
  • Stand-in address space throughout. A reconstruction, not any operator's configuration, and not a claim that the outage would have been prevented.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

EDGE-1 as built: the peer group resets on a malformed attribute

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_error_tolerance_off. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • EDGE-1
  • 192.0.2.1
  • 64510

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

the healthy variant: bgp-error-tolerance on the peer group too

now: pass

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_error_tolerance_off. Pinned status: pass, the check finds nothing.

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: As with the 2023 case, only the presence of tolerant error handling on eBGP sessions is auditable.

Not fixed: RFC 7606 error handling on other vendors

Junos bgp-error-tolerance is parsed and audited; other vendors' behaviour is a release default with no config line, so the audit answers NULL there.

Recorded against: Juniper 2023, OTC 2025

Imported from docs/replays/otc-attribute-2025/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.