Haiku caching, a backfill outgrowing its cap, and what's still open
Value
3.0
■■■□ of 4
Time
32m
active
Elapsed
1h 32m
wall-clock span
Commits
0
from this session
Cost
$8.91
model spend
Tokens
71k
uncached in, plus out
Tool calls
135
by the agent
Outcomes
10
7 left open
Summary
Written by the summariserRob asked whether the enrichment client used prompt caching after the move from DeepSeek to Haiku, whether article drafts run on Haiku and were evaluated, and then wanted every conversation backfilled with Haiku at higher parallelism.
Rob asked whether the enrichment client used Anthropic prompt caching after moving to Haiku, and the agent found that no request sent cache_control. Measuring the 52 Haiku calls since cutover showed caching would have saved about $0.004, so Rob chose to measure first and the agent recommended leaving it off. The agent then confirmed that summaries, drafts and redaction all run on Haiku, and that the October eval graded article text, with a rewritten draft prompt scoring best but unshipped. Rob chose to ship that prompt and backfill all history, including redrafting the nine conversations that already had entries. The backfill was capped at four concurrent jobs by a hard-coded value, so after Rob chose sixteen the agent added a setting, deployed it, and restarted the run, whose first hour looked healthy. The run was still going when the transcript ends.
What came out of it
- Confirmed no Anthropic cache_control in the enrichment code; no caching change made, and the caching recommendation was to leave it off.
- Measured 52 Haiku calls since the 2026-10-07 cutover costing $0.146 in total, with no retries; caching would have saved about $0.004.
- Census: 513 settled in-window conversations already summarised; 1,686 older candidates outside the 30-day window.
- Confirmed Camden sets the model to claude-haiku-5-5 for summaries, drafts and redaction.
- Shipped [commit] (pushed and deployed to Camden): automatic drafts use draft-v6, owner redrafts use draft-v7 with notes, and the personal-approval redaction hold covers both v5 and v7. Readiness returned 200.
- Fixed a CodeRabbit finding so the owner-notes test checks the user message and fails when notes are dropped.
- Shipped [commit] (pushed and deployed): AJV_ENRICH_CONCURRENCY setting, default 4, set to 16 on Camden, replacing two hard-coded caps; added a concurrency test.
- Appended concurrency notes to Backlog task AJV-0156.
- Full-history backfill accepted for 2,292 conversations; restarted at concurrency 16 after 258 had completed, leaving 2,034 to run.
- First-hour results: 280 summaries valid on first attempt; 6 drafts failed validation then passed on retry; one personal article's redaction failed twice on the public-identifier check and is held for review.
After moving the enrichment work from DeepSeek to Haiku, I wanted to know whether I was still getting prompt caching. I wasn't. None of the requests sent the cache marker, so every call paid full input price. Before changing anything, I measured. The 52 Haiku calls since the cutover cost $0.146 in total, and caching would have saved about $0.004. Not worth the complexity, so I'm leaving it off. Most of the bill is output, and the real lever would be shorter summaries, which is a separate question.
The bigger question was whether every conversation had a summary and draft. Only the recent window had been covered, leaving 2,292 conversations in the full history waiting. I decided to backfill them. Along the way I shipped a rewritten draft prompt that scored best in the October article eval. There are two versions now: one for automatic drafts and one for my own redrafts, which take notes. A test caught a case where those notes were being dropped, and I fixed that.
The backfill was only running four conversations at a time, because of a hard-coded number in the code. There were actually two caps, and raising just one wouldn't have helped. I replaced both with a single setting, defaulting to 4, and set it to 16 on Camden. I added a test that counts how many calls reach the gateway at once, deployed, and restarted the run. It picked up the remaining 2,034 conversations without redoing anything.
In the first hour, all 280 summaries passed validation on the first try. Six drafts failed and then passed on retry. One personal article failed redaction twice on the public-identifier check, so it's held for me to read. That's the system doing what it should.
Still open: the backfill is running, so I haven't checked the final counts. Nine conversations that already had entries need redrafting once it finishes. I also expect a large approval queue, likely over a thousand drafts, which I haven't read yet. I haven't confirmed that my Anthropic account can handle 16 at once, or whether the gateway passes caching through. An API key also ended up in a stored prompt during the eval, so I need to check whether it needs rotating.
Left open
- The backfill was still running when the transcript ends; final done and failed counts are not yet checked.
- Redrafts for the nine conversations with existing entries were chosen but not yet run; they are to be handled after the backfill finishes.
- One personal article is held after failing redaction twice and needs Rob's review.
- Expect a large approval queue, likely over a thousand new drafts; none reviewed yet, and article quality was checked only by query, not by reading the text.
- Backlog task AJV-0156 remains open pending the backfill finishing and the entry-conversation redrafts.
- Anthropic API key appeared in an eval prompt now stored in the agent-history catalogue; the agent suggested rotating it if still live, and its status is unknown.
- Prompt caching was not implemented; whether the Cloudflare AI Gateway passes cache_control through was not tested, and Anthropic account tier was not confirmed, so the 16-concurrency headroom is an estimate.