Measuring tokens per second across pi loops
Value
2.6
■■■□ of 4
Time
17m
active
Elapsed
28m
wall-clock span
Commits
2
from this session
Cost
$2.28
model spend
Tokens
22k
uncached in, plus out
Tool calls
27
by the agent
Outcomes
5
6 left open
Summary
Written by the summariserFind average tokens per second for pi loops by model, with a fast-tier filter, and get the agent-history dashboard updated. Then research other request and response metadata worth capturing.
Rob asked for average tokens per second across pi loops by model, including fast-mode runs, to see whether OpenAI's speed improvements had shown up. The catalogue has no per-call duration, so the agent estimated it from the start and end timestamps in the raw pi transcripts on this Mac, covering about 34k calls. The preview suggested standard-tier speed had not improved and that fast mode ran roughly 1.5 to 2x faster, but the window was short, came from one machine, and the model-by-day split was uneven. Rob asked for backlog items rather than an immediate dashboard change, so no parser, schema, deploy or panel work was done. Three read-only research passes then proposed candidate fields from pi, Claude Code and Codex transcripts, and Rob opted into 16 of them, each logged as a backlog task.
What came out of it
- Preview tokens-per-second table from raw pi transcripts on this Mac (about 34k calls), overall and median by model and service tier. Duration is end minus start time, so it includes time to first token. The tier is only recorded from 2026-10-03.
- HRN-0243 logged: store per-call duration for pi LLM calls and add a tokens-per-second dashboard panel.
- HRN-0244 to HRN-0259 logged (16 items, opted into): estimated call duration for Claude and Codex; Claude thinking time; pi first-token time, retries and response headers; Codex fast tier into service_tier; pi per-call cost; Claude cache-miss reason; Claude input rewrites; unreliable cost flag; pi stop reasons and error types; pi effort per call; Claude advisor, region and iterations; Codex message phase and reasoning-summary mode; Codex trace id and root turn; Claude turn origin and pending background work; pi tool result details and subagent limits; Codex MCP plugin and read-only flag.
- Backlog task commits made in the ~/.agents repo, not pushed.
- Rob chose the Codex MCP item but left out the credits balance.
I wanted to know whether OpenAI's recent speed improvements had actually reached my own agent runs. The question sounded simple, but it hit a gap straight away: my catalogue records when each model call happened, not how long it took.
The raw pi transcripts do hold the answer. Each assistant message has a start and an end timestamp, so I could work out a rough duration per call and divide output tokens by that time. The result includes time to first token as well as generation, so it measures end-to-end speed rather than pure generation. Fast mode is recorded too, but only from 3 October, so earlier calls show up as unrecorded.
On this Mac, across about 34k calls, fast mode looked roughly 1.5 to 2 times quicker than the standard tier on the same model. For gpt-6.1-sol the median was around 44 to 53 tokens per second in fast mode, against 26 to 27 on the default tier. The standard-tier numbers hint that normal speed hasn't moved, but I wouldn't lean on that yet. The window is short, it comes from one machine, and the split by model and day is uneven. The tier label only starts on 3 October, so some of the earlier figures may already include fast runs.
I didn't touch the dashboard. A proper panel means storing a duration for every call, a parser change and a redeploy to the homelab, and I wanted to look at what else was worth capturing first. Going through the pi, Claude and Codex transcripts turned up a long list of candidate fields. I picked sixteen and logged them as backlog tasks, alongside the duration work, seventeen in total. They cover things like per-call cost, why a Claude prompt cache was missed, Codex's fast tier being written into the catalogue, and stop reasons and error types for pi.
Still open: the homelab deploy is waiting on my go-ahead. Pi can't report time to first token yet, so that needs a change to the harness itself. The Claude and Codex durations would be estimates, and Codex already records first-token time per turn.
Left open
- No dashboard panel, parser change, schema migration or homelab deploy was made. The deploy is still waiting for Rob's go-ahead.
- The model-level trend is not firmly supported. The earlier table put gpt-6-sol around 40 tok/s on 09-28/29 and gpt-6.1-sol on the default tier around 26 on 10-04/05. The closing summary restated gpt-6.1-sol and fast-tier figures that do not match that table, so treat the trend as unconfirmed.
- Tier data only starts on 10-03, so some 09-29 figures may include fast-mode runs.
- The preview covers this Mac only. Runs from the other machine and the 09-30 to 10-02 window are missing.
- Time to first token for pi needs the harness change in HRN-0246. Claude and Codex durations are estimates that can include queue or hook time.
- The agent left an uncommitted edit in HRN-0240 alone, noting it was not its own.