All issues

harness-digest · 2026-W43

Week 43from 19 Oct 2026

Most of this week's items argue that agents get better when you give them less to do and a clearer way to check their work. Several pieces shrink the context an agent carries: a thin router, a shorter CLAUDE.md, and small composable skills. Others treat verification as the real finish line, whether that means tuned eval thresholds, evidence-based code review, or an agent that runs its own test loop. The cost piece shows the same discipline from the budget side: measure what each session actually sends before trying to optimize it.

scan
1
passed the judge
20
picked
7

30 web searches, 1.6M tokens read. Every run writes a log: see them on GitHub.

1repo · 8 min · harness design

9/10

mattpocock/skills

Agents misread intent, ramble, ship broken code and pile up complexity; a set of small, composable skills adds grilling sessions, a shared glossary, TDD loops and design checks to counter each failure.

How it works

  • Run /grill-me or /grill-with-docs first so the agent interviews you until every branch of the plan is settled.
  • Record project jargon in a GLOSSARY.md and ADRs, so the agent uses one precise term instead of long phrases.
  • Use the /tdd skill for red-green-refactor, writing a failing test first so the agent always gets feedback.
  • Use /diagnosing-bugs to build a red feedback loop, then minimise, hypothesise, instrument, fix and add a regression test.
  • Run /code-review as parallel sub-agents, one checking repo standards and one checking spec fidelity, so neither contaminates the other.

Steal this: Before your next feature, have the agent grill you with a /grill-with-docs style interview and write the agreed terms into a GLOSSARY.md it reads in later sessions.

Go deeper if you're running coding agents daily and want to borrow a tested, editable set of skill files rather than adopt a heavy process framework.

“The fix for this is a grilling session - getting the agent to ask you detailed questions about what you're building.”

Quote checked against the source by the verifier.

2blog · 12 min · real setups · 19 Oct

9/10

Cutting our inference bill by 68% by being obsessive about optimization

An AI code reviewer's cost per review passed $2 as models and context grew; logging each session's first request, then fixing cache prefixes and call counts, cut it 68% without losing real findings.

How it works

  • Log token usage and a hash of the system block for each session's first LLM call to see cache misses.
  • Split the system prompt at a marker so shared tool and agent text gets its own cache breakpoint.
  • Strip repository-specific content from before that breakpoint, and put reference files in the prompt instead of tool reads.
  • Remove tools reviewers can't use, read files in parallel, skip reviewers with nothing new, and stop after the review posts.
  • Check quality by comparing findings authors acted on, and by blinded model judges, before moving models or effort.

Steal this: Before your next agent cost review, log a hash of each session's first system block plus its cache read/write tokens to find whether shared prefixes are actually hitting the cache.

Go deeper if you're running agents at volume and want a worked example of cache-prefix fixes and checking that cheaper settings still find real problems.

“the two things we’d check first are whether each session’s first request actually reads from the cache”

Quote checked against the source by the verifier.

3blog · 12 min · verification · 20 Oct

8/10

How To Use Decision Models for Evals

Cheap decision-model judges are often run at the default 0.50 threshold, which can waste performance; tune the threshold on labeled traces, then check calibration and cost before choosing a model.

How it works

  • Label a set of traces pass/fail, then save each model's failure probability for every trace.
  • Set minimum precision and recall for your use case, such as 80% each, and rule out models that miss them.
  • Tune each model's threshold with stratified five-fold cross-validation, picking on training splits and measuring on held-out traces.
  • Check calibration with a reliability plot and Brier score; apply Platt scaling only if it measurably improves them.
  • Compare tuned F1 against cost per 100 eval runs to find the cheapest model that clears your bar.

Steal this: Before you next adopt a cheap LLM or decision-model judge for automated evals, save its scores on labeled traces and ask a coding agent to cross-validate a threshold against minimum precision and recall instead of using 0.50.

Go deeper if you're replacing an expensive LLM judge with a cheaper classifier and want a rigorous way to compare thresholds, calibration and cost.

“Using this default without checking it against your data can give you subpar results.”

Quote checked against the source by the verifier.

4blog · 6 min · harness design · 19 Oct

8/10

The Thin Orchestrator: How We Made Our AI Agents More Reliable by Giving the Router Less to Do

One agent loaded with all domain knowledge started hallucinating and mixing up subsystems; a thin deterministic router that delegates reasoning to isolated, specialized subagents made the system far more reliable.

How it works

  • A deterministic orchestrator classifies intent, gathers fields, checks reliability gates, and delegates, but never reasons about code itself.
  • Fourteen subagents each hold only their own domain knowledge, so one data source's patterns can't leak into another's.
  • Every request follows a fixed 7-step protocol; only the delegation step uses LLM reasoning, inside an isolated subagent.
  • Workflow templates sequence the same subagents in different orders, so new use cases need no new agent code.
  • Knowledge files load on demand at runtime, so updates take effect in the next session without redeploying the orchestrator.

Steal this: Before your next multi-step agent build, split every component into routing (deterministic code) or reasoning (isolated subagent), and give each subagent the same instruction-file template: knowledge sources, required inputs, steps, troubleshooting.

Go deeper if you're running one big agent with a sprawling context that confuses concepts across subsystems, and want a concrete structure for splitting it up.

“The agent had everything it needed. That was the problem.”

Quote checked against the source by the verifier.

5blog · 6 min · harness design · 19 Oct

8/10

Your CLAUDE.md Is Too Long: What to Cut So Claude Code Follows It

Bloated CLAUDE.md files get their rules ignored, so cut every line that fails a simple test and move the rest into hooks, skills, path-scoped rules or a local file.

How it works

  • Ask of every line: would removing this cause Claude to make mistakes? If not, delete it.
  • Delete personality prompts, generic advice and folder tours that Claude can learn by reading the code.
  • Move formatting rules into a PostToolUse hook that runs prettier after every Edit or Write.
  • Move multi-step procedures into skills, and area-specific rules into .claude/rules files with a paths field.
  • Put personal settings in a gitignored CLAUDE.local.md, then check /context and run /doctor prompt-audit.

Steal this: Before your next change to CLAUDE.md, run the one-line test on every line and move style rules into a formatter hook, migration recipes into a skill, and API rules into a path-scoped rule.

Go deeper if you're watching Claude Code ignore rules in a CLAUDE.md that has grown past a couple hundred lines and want a concrete pruning checklist.

“Would removing this cause Claude to make mistakes?”

Quote checked against the source by the verifier.

6blog · 11 min · lighter review · 16 Oct

8/10

Code Review After Code Gets Cheap

Agent-written pull requests outpace human reading time, so make evidence a condition of acceptance: agree on behavior and blast radius first, then spend review effort where evidence is weak or consequences are serious.

How it works

  • Before generation, agree the intended behavior, boundary examples and expected blast radius with whoever owns the requirement.
  • Ask an agent to compare the spec with the changed files and callers, flagging missing or unexplained behavior.
  • Check test assertions against the agreed rule; remove the guarded check temporarily to confirm the test actually fails.
  • Attach a short review record: agreed behavior, linked evidence for this revision, extra behavior, blast radius, open questions.
  • Scale inspection to blast radius; spot-check changes treated as routine and strengthen checks if gaps keep appearing.

Steal this: Before approving your next agent-generated pull request, require a short review record and temporarily disable the key guard clause to prove its test fails.

Go deeper if you're the bottleneck reviewing agent output and want a policy for deciding which changes deserve line-by-line reading.

“Treat blast radius as a review criterion on its own.”

Quote checked against the source by the verifier.

7blog · 8 min · real setups · 15 Oct

8/10

How Meta Built Forum With AI-Native Engineering

Agents produce plausible but unverified code, so Meta's Forum team made verification the finish line and changed the codebase and tooling so the agent runs its own plan-build-test loop.

How it works

  • Count a task as done only when automated tests and visual checks pass, not when it compiles.
  • Rules, skills, commands and hooks encode conventions; a repeated agent mistake gets fixed once as a permanent rule.
  • Give the agent tools to read the live view hierarchy and list on-screen elements before acting, instead of describing the app.
  • Run repeatable stress scenarios (fast scrolling, rapid navigation) so the agent can spot anomalies, reproduce, fix and verify.
  • Attach build results, test outcomes and recorded verification sessions to each pull request so engineer sign-off is fast.

Steal this: The next time your agent edits the wrong thing, add a tool that lets it inspect real state (like the live UI tree or a config dump) before acting, instead of adding more prompt description.

Go deeper if you're moving from autocomplete-style AI use to agents running full build-test loops and want concrete harness, verification and context practices.

“What improved quality wasn’t better prompting. It was context.”

Quote checked against the source by the verifier.

The judge’s reasons were first logged in week 45, so this issue only has its scores. This issue as markdown, as the harness committed it.