Moving n8n's summaries from DeepSeek to Claude Haiku 5.5
Value
3.4
■■■□ of 4
Time
25m
active
Elapsed
1h 00m
wall-clock span
Commits
0
from this session
Cost
$6.94
model spend
Tokens
67k
uncached in, plus out
Tool calls
87
by the agent
Outcomes
11
5 left open
Summary
Written by the summariserEvaluate whether three n8n workflows running on deepseek-flash should move to Claude Haiku 5.5 at different effort levels, then migrate them and tune their prompts.
Rob asked whether the n8n workflows on deepseek-flash should move to Claude Haiku 5.5. The agent replayed 20 real calls from the AI Gateway logs at medium, high and xhigh effort and graded all 80 outputs itself, blind. Haiku beat deepseek-flash on the two release workflows and was slightly worse on community triage, and the extra effort levels added little. Rob then chose to move all three workflows, keeping meeting notes on direct Anthropic, and asked for prompt tuning. The agent rewrote the prompts and compared old and new versions on 43 past inputs, with the rewrite preferred in most cases and the largest gain on triage. It then minted a dedicated gateway token, created an Anthropic credential and switched the three published workflows to Haiku 5.5 at high effort, but no live production run had been observed when the session ended.
What came out of it
- Haiku 5.5 vs deepseek-flash on 20 replayed calls (blind main-thread grading): n8n release notes 3.71 for deepseek vs 4.14 (medium), 4.29 (high), 4.29 (xhigh); release-PR summaries 4.12 vs 4.44 at all levels; community triage 4.00 for deepseek vs 3.70, 3.80, 3.80 on 5 cases.
- All 80 eval outputs passed parse, schema, length-cap and key-echo checks; per-call cost was about $0.0004 to $0.0007 for every variant.
- Prompt rewrite vs current prompt on Haiku high (43 cases, blind pairwise grading by an Opus subagent): release notes 4.10 to 4.50 (rewrite preferred 7, tie 3, current 0); PR summaries 4.29 to 4.79 (9/3/0); triage 3.48 to 4.71 (14/4/3). Rewrite cut PR and triage output tokens by about a third.
- Published n8n workflows AVK6R9L7CxwkCzS3 (release notes), K8xQrV2LCiVi0GSJ (community triage) and tBOdQV367X1KleGi (release-PR summaries) now use an Anthropic Chat Model node with claude-haiku-5-5, adaptive thinking, effort high and the rewritten prompts; the meeting-notes workflow was left unchanged.
- Created a Cloudflare token n8n-ai-gateway-run with gateway Run scope only, stored at credentials/cloudflare/ai-gateway/n8n-token, and an n8n Anthropic credential fMZ68UqStQ6T93QF that points at the gateway with BYOK.
- Kept the old DeepSeek credential UNEWR6XZU4EqVJNz as an unused rollback.
- Probes showed a placeholder provider key returns 401 through the gateway, while the gateway token itself works as the key.
- Found no community n8n node for Cloudflare AI Gateway.
- Documentation written: n8n/reference/llm-gateway.md (rewritten gateway reference), the n8n AGENTS.md credential table, three n8n workflow docs, and cloudflare/ai-gateway/README.md with the key-header finding.
- Eval scripts, prompts and grades archived in n8n/evals/haiku-2026-10/ for re-running.
- Answered the caching question: Anthropic prompt caching stays off, since most prompts are below Haiku 5.5's 512-token minimum and calls are too far apart to benefit.
I wanted to know whether the three n8n workflows running on DeepSeek's flash model should move to Claude Haiku 5.5, and whether the Cloudflare AI Gateway I already use could carry the traffic. They cover n8n release notes, summaries of release PRs and triage of community issues. Two of them don't keep successful runs, so I pulled 20 real calls from the gateway's request logs and replayed each one against Haiku at medium, high and extra-high effort. I graded all 80 outputs myself, blind, with shuffled labels.
Haiku came out ahead on both release workflows. Release notes went from 3.71 on DeepSeek to 4.14 at medium and 4.29 at high and extra-high. Release-PR summaries went from 4.12 to 4.44 at every level. Triage was slightly worse, 4.00 against 3.70 to 3.80, though that was only five cases. The extra effort bought almost nothing, every output passed the schema and length checks, and the cost stayed well under a tenth of a cent a call.
Next I rewrote all three prompts following Anthropic's guidance for Haiku 5.5: a one-line description of the reader, XML tags around the data, a few made-up examples, and no reasoning instructions, since effort handles that. On 43 past inputs the rewrites won most comparisons. Triage improved most, from 3.48 to 4.71. The old prompt often told me to fix bugs in other people's repos and rated other people's me-too comments as critical. The rewrite reserves the top priority for breakage in my own repos.
I've now switched all three workflows to Haiku 5.5 at high effort, through the gateway, using a new token scoped only to running it. The meeting-notes workflow still calls Anthropic directly and is unchanged. The old DeepSeek credential is still there as a rollback.
Still open:
- No live production run has been checked yet. A watcher is polling for the first one.
- I set high effort, but the eval only needed medium. I need to confirm that's what I want.
- I haven't confirmed the n8n node version sends the effort setting until a run goes through.
- The samples are small, and the first round was graded by me alone.
I also checked prompt caching and left it off. The release prompts fall below Haiku's 512-token minimum, and the triage calls are too far apart for the five-minute cache to help.
Left open
- No live production run of the migrated Haiku workflows had been seen; a background watcher was polling gateway logs for up to 3 hours.
- Rob's reply 'All 3 but Haiku high please' is ambiguous. The migration went ahead at high effort, while the first eval found medium sufficient, so Rob should confirm the setting.
- Not verified that the n8n node version (2.41.3) sends the effort setting in a live run.
- Samples are small: triage had 5 cases in the first round and 21 in the prompt round, and the first round was graded by one grader.
- Changes were not committed in this session; the sync worker is expected to commit them.