Chasing stream_incomplete errors through codex-lb and pi
Value
3.4
■■■□ of 4
Time
5h 10m
active
Elapsed
76h 25m
wall-clock span
Commits
17
from this session
Cost
$269.52
model spend
Tokens
3.5M
uncached in, plus out
Tool calls
922
by the agent
Outcomes
13
9 left open
Summary
Written by the summariserRob asked why codex-lb sessions were hitting stream_incomplete and owner-unavailable 502s, whether his open PR #2467 fixed them, and then wanted diagnostics, failover fixes, and a pi retry and upgrade path.
Rob's unmerged PR #2467 turned out not to cover the observed 502 transport path, so diagnostics were added on a local branch and deployed to Camden, along with a Grafana alert that reached IRM. Those diagnostics showed streams terminated when a 64-message websocket queue overflowed, which led to queue, account-health, quota-reset and fresh-start recovery changes deployed to Camden with 385 tests passing. After the first fix, roughly 98.9% of about 924 recorded outcomes succeeded, though occasional interruptions remained. The work then moved to pi: its retry budget was bounded to 20 minutes, pi was upgraded to 1.1.0 on both Macs, and a later recovery fix was installed on both. The most recent investigation found that brewmdm failures were upstream cyber_policy refusals that pi mislabelled as recovery failures; a fix is proposed but not implemented.
What came out of it
- PR #2467 assessed as covering a dead-anchor 404 path rather than the observed 502 websocket path; not deployed.
- Local branch diagnostics/native-websocket-failures, commit [commit], deployed to Camden via update.sh; nothing pushed upstream.
- Grafana alert 'Codex LB active stream interrupted' added, excluding idle closes; it fired on real failures and reached IRM.
- Identified consumer_backpressure failures from a 64-message websocket queue overflow holding about 12.7 KB.
- Commit [commit] deployed: queue-overflow diagnostics, six new Prometheus metric families, and a codex-lb runtime dashboard covering 71 metric families with 110 queries validated.
- Commit [commit] deployed: per-account limits set to 4 response creations, 8 streams and 1 recovery slot; burst-tolerant bounded websocket queue; local overflow no longer penalises account health; exhausted weekly quota reset preserved; fresh-start recovery for requests that can run independently on another account. 385 tests passed.
- Five stuck sessions had stale bindings marked abandoned after backups; three moved to other accounts and completed requests.
- Commit [commit] deployed to Camden, healthy, with 11 successful requests in the short post-deploy window.
- loop-pi recovery work: 20-minute outage budget, retry of interrupted local-tool responses without re-executing tools (commits [commit] and [commit]); HRN-0316 completed.
- pi upgraded from 1.0.4 to 1.1.0 across the eight pi-family packages; adoption lock 2e7d714 published; live root/child canary passed in 43 seconds; GFMBP and MBP16 upgraded with previous builds retained.
- agent-workflows 1.18.5 published, removing the whole-Mac deferral requirement from the pi-upgrade skill.
- LiteLLM assessed: all three routes point to the same codex-lb backend, so it adds no independent capacity; recommendation was to keep the current route.
- Brewmdm failures identified as upstream cyber_policy refusals misclassified by pi as unrecognized, with a fix proposed.
My pi loops kept hitting stream_incomplete errors and a 502 about a rejected response anchor. The harness usually retries until it succeeds, so these were easy to miss. I had an unmerged codex-lb PR (#2467) and wanted to know whether it would help.
It doesn't. #2467 fixes a related recovery bug that causes repeated 404s, and the 502 comes from a different guard that the PR doesn't touch. The logs couldn't show why the upstream websocket dropped, so I built diagnostics on a local branch, deployed them to Camden, and added a Grafana alert for interrupted active streams. Idle closes are excluded. The alert fired on real failures and reached my incident notifications. Phone delivery is still unverified.
The diagnostics pointed at a 64-message websocket queue overflowing and cutting streams off. I changed codex-lb to cope: per-account limits, a burst-tolerant queue, local overflow no longer counting against an account's health, exhausted weekly quota resets preserved, and fresh-start recovery where a request can run independently on another account. That passed 385 tests and was deployed to Camden. Afterwards, about 98.9% of roughly 924 recorded outcomes succeeded. Some interruptions remain, and failover is not yet proven.
On the pi side, I capped the outage retry budget at 20 minutes, added retries for interrupted local-tool responses without re-running tools, and upgraded pi from 1.0.4 to 1.1.0 on both Macs.
The latest investigation changed the picture. The brewmdm failures weren't disconnects at all. They were upstream policy refusals, which pi relabelled as unrecognized and reported as a recovery problem. I've proposed a fix that shows refusals accurately and limits automatic retries to transient failures. It isn't implemented yet. I also considered LiteLLM in front of codex-lb and kept the current route, since all its routes lead to the same backend and add no independent capacity.
Still open: why the original disconnects happened and why the queue's consumer lagged; the occasional stream errors; the pi fix; and a telemetry pipeline capturing process arguments that can contain credentials, which needs redacting and the affected credentials rotating.
Left open
- Root cause of the initial websocket disconnects remains unproven; which queue filled and why its consumer lagged was not established.
- Occasional stream_incomplete and failover errors continue; a quota scenario did not yet prove an account switch, so failover is not established as fixed.
- Phone delivery of the Grafana alert is unverified.
- Fix for pi's cyber_policy misclassification is proposed and awaiting Rob's choice; no code changed.
- GFMBP Personal Codex browser-trust failure remains open as HRN-0318.
- Traefik telemetry exposing credential-bearing process arguments still needs redaction and credential rotation; not done.
- Existing in-flight pi sessions keep their old builds until restarted, and some still need manual resume messages.
- Native WebSocket pi transport remains experimental and off by default.
- codex-lb PRs #2467 and #2486 remain open and unmerged; LiteLLM not adopted.