All issues

harness-digest · 2026-W44

Week 44from 26 Oct 2026

Most of this week's items replace an agent's own say-so with something that can be checked: screenshots captured automatically, evidence required in every report, stop hooks, a fresh-context reviewer, and guardrails enforced in code at the tool-call layer rather than in the prompt. The same idea carries into measurement and memory, with eval datasets scored per case against a baseline and answers tied to an evidence graph of typed citations. The practical takeaway is to put a mechanical check between the agent and its claim of "done," and to run it on every change.

scans
2
passed the judge
28
carried over
3
picked
7

46 web searches, 1.9M tokens read. Every run writes a log: see them on GitHub.

1repo · 6 min · verification

9/10

feat(skills): add ui-evidence for before and after UI screenshots

Hand-taken UI screenshots are flaky and tie up shared dev stacks, so a skill builds isolated before and after copies of the app and captures light and dark screenshots automatically.

How it works

  • `up` builds a git ref in a detached worktree, then starts a labeled Postgres container and the API on free ports.
  • A scenario file seeds users and sessions through SQL and everything else through the API; RPCs missing on the old ref are reported, not faked.
  • `shoot` uses headless Chromium, sets the theme, waits for network idle and animations, and writes names like shot-before-dark.png.
  • A missing selector or an unapplied theme fails the shot, and its screenshot goes to a failed/ folder instead of the evidence.
  • `table` builds the PR-body table, `gh pr edit --attach` uploads the images, and `down --all` cleans up containers and worktrees.

Steal this: Next time an agent changes a UI, wrap screenshot capture in a skill that forks into a subagent, fails loudly on missing selectors, and returns only paths and failures to the main session.

Go deeper if you're reviewing agent-made UI changes and want a repeatable, isolated way to attach before and after proof to every PR.

“A selector that never appears, or a theme the page did not apply, fails the shot.”

Quote checked against the source by the verifier.

2blog · 7 min · verification · 25 Oct

9/10

Make coding agents prove their work

An agent's own summary describes what it meant to do, not what happened, so require checkable evidence in every report and enforce the fast checks with a Stop hook.

How it works

  • Run checks cheapest first: typecheck, lint, build with prerender, bundle budget, then screenshots at desktop and phone widths.
  • Every report must include changed files, pasted command output, test counts, budget numbers, screenshot paths, and a section on what was not checked.
  • A Claude Code Stop hook runs typecheck and lint; exit code 2 returns errors to the agent so it keeps working.
  • The hook checks the stop_hook_active flag and allows one forced retry, avoiding endless loops on unfixable errors.
  • A CI budget script gzips the initial JS and CSS, fails above 250 KB, and prints numbers on every build.

Steal this: Before your next agent task, add a Stop hook that runs typecheck and lint and exits 2 on failure, and require the report to end with a 'what I did not check' section.

Go deeper if you're merging agent changes on trust and want the full hook script, budget check and Playwright screenshot script to copy.

“An agent's summary of its own work is the least reliable part of its output”

Quote checked against the source by the verifier.

3repo · 10 min · harness design

8/10

harness-template

Agents drift without a stated goal, state, or independent checks, so this template installs a goal, one-item backlog, boot hook, guard hooks and a read-only reviewer subagent in one command.

How it works

  • A session-start hook injects the goal, current backlog item and git state so the agent always begins from real state.
  • The /ship skill takes one backlog item: plan, human OK, build, read-only reviewer subagent checks, then record and commit.
  • A Stop hook runs the project's check script and sends the agent back to work if the check is red.
  • A Bash guard hook refuses force-push, reset --hard and rm -rf, and asks before migrations and deploys.
  • The reviewer has no write tools, and non-blocking findings become bug or debt items in the backlog.

Steal this: Add a Stop hook that runs your test command and blocks the agent from finishing while it fails, the next time you set up a Claude Code project.

Go deeper if you're building your own agent harness and want working hook scripts, a maker/checker split and a way to update it over time.

“Every file here encodes an assumption about what the model can't do.”

Quote checked against the source by the verifier.

4repo · 6 min · harness design

8/10

agent-harness-template

Agents start each session blind, claim "done" without running anything, and review their own code. This template swaps longer prompts for rule files, a single check command, stop hooks and a fresh-context reviewer.

How it works

  • One AGENTS.md holds hard rules with reasons, a code map and commands, and every tool reads it.
  • A stop hook runs quick checks on changed files, and the agent can't finish until make check passes.
  • PROGRESS.md and docs/decisions.md carry state between sessions; a handoff skill updates them and a start hook displays them.
  • A reviewer subagent reads only the diff in a fresh context against REVIEW.md, for at most two rounds.
  • A fill prompt mines past agent sessions, turning repeated corrections into rules and checks.

Steal this: Next time an agent repeats a mistake twice, add a failing check to your single check script instead of another paragraph in the instructions file.

Go deeper if you're setting up a repo for Claude Code or Codex and want a working set of hooks, skills and review files to copy.

“A mistake that happened twice becomes a check in scripts/check.sh, not another paragraph.”

Quote checked against the source by the verifier.

5repo · 4 min · verification

8/10

themis: a customer support agent with code-enforced guardrails and an eval harness

Prompt-only rules let a support agent approve refunds it shouldn't, so this repo enforces limits in code at the tool-call layer and measures behaviour with a 45-scenario eval suite.

How it works

  • Support procedures live as markdown files in an aops folder, and the agent loads only the one it needs.
  • Every tool call passes through a guardrails module that checks identity verification and order ownership.
  • Refunds over $150 are routed to a human for approval rather than decided by the model.
  • A live trace shows the current procedure step, tool calls and each guardrail decision.
  • 45 scenarios (normal, edge, attacks like prompt injection) are checked against backend state and graded by an LLM judge.

Steal this: Before your next agent ships, move one hard rule (like a spending limit) out of the prompt into a code check on the tool call, then run an eval that compares both versions.

Go deeper if you're building an agent that takes real actions, like refunds, and need evidence that code guardrails beat prompt instructions.

“With the guardrails in code it was $0.”

Quote checked against the source by the verifier.

6blog · 9 min · verification · 28 Oct

8/10

Hill-Climbing Agent Quality with Agents CLI Evaluation

Agents that work in a demo drift after prompt, tool or model changes, so run a small eval dataset on every change and compare per-case scores against a baseline.

How it works

  • Write a small dataset of about four cases covering different behaviors, so it runs in under a minute on every change.
  • Generate agent traces, then grade them with an LLM-as-judge metric, and save the full result JSON, not just the mean.
  • Diff the baseline and candidate result files per case to see where a change helped or hurt.
  • Run a nightly job on a bigger regression set, with a script that exits non-zero if the mean drops past a threshold.
  • Keep the dataset (inputs) and eval config (expectations) in separate committed files so you know which half changed.

Steal this: Before your next prompt or model swap, run the same small eval set before and after, then compare per-case scores rather than the mean to see exactly which behavior moved.

Go deeper if you're swapping prompts or models on an agent and want a concrete eval loop with regression alerting, including worked before-and-after numbers.

“Snapshot result files, not scores.”

Quote checked against the source by the verifier.

7blog · 8 min · harness design · 28 Oct

8/10

How to build evidence-grounded agent memory with OpenWiki and Graphs

Agents answering questions across many documents rebuild context from scratch and cite inconsistent evidence; a compiled wiki plus a deterministic evidence graph of typed citation edges makes answers more accurate and repeatable.

How it works

  • Compile source documents into linked wiki pages where each claim cites source lines and a hash of those lines.
  • When a source changes, the hash changes, so stale claims are flagged without a model call.
  • Split documents into uniquely identified paragraphs so claims point to exact passages, not whole documents.
  • Materialize citations as typed edges such as cites, refers_to and supersedes, with no extra model calls.
  • Give the agent two tools: locate finds a starting node, expand follows typed edges and stops at branches.

Steal this: Before your next agent works across a pile of policies or docs, turn its existing citations into typed edges (refers_to, supersedes) and expose simple locate and expand tools for following them.

Go deeper if you're building agents that reason across many linked documents and need reproducible, auditable answers, or you're weighing whether memory is worth its cost.

“The graph’s job is evidence navigation, not modeling everything the organization knows.”

Quote checked against the source by the verifier.

The judge’s reasons were first logged in week 45, so this issue only has its scores. This issue as markdown, as the harness committed it.