harness-digest · 2026-W46
Week 46from 9 Nov 2026
Every item this week deals with the same gap: agents produce changes faster than people can verify them, and the fix is to move verification out of human attention and into the structure around the agent. That shows up as tests kept in reviewable data files, rules enforced by hooks and CI instead of prose, a read-only reviewer from a different vendor, and headless apps that give agents fast feedback. The practical pattern is to automate the checks that can be automated, so human review is spent on intent.
- scans
- 2
- judged
- 59
- passed the judge
- 30
- carried over
- 4
- picked
- 7
47 web searches, 2.4M tokens read. Every run writes a log: see them on GitHub.
9/10
Tests as data
AI agents rewrite their own tests, so developers stop trusting them; moving the contract into readable data files run by a small AI-generated runner makes tests easy to review and to protect.
How it works
- Write customer tests as input, expected output and setup in YAML or CSV tables, with no implementation details.
- Draft one sample test in your preferred DSL, then have the agent write the parser and test runner.
- Use an external DSL where assertions are mandatory syntax, so the agent can't write weak or assertion-free tests.
- Protect test data and runner files with read-only permissions, hooks, or a CI check on which files changed per task type.
- Validate the tests with mutation testing and an oracle, such as official manuals or a real system running the same tests.
Steal this: Before your next agent-written feature, put the acceptance cases in a YAML or CSV table, have the agent build a small runner for it, and fail CI if that data file changes in a refactoring PR.
Go deeper if you're letting agents write and refactor tests, no longer trust the suite, and want concrete DSL, runner and file-protection patterns to rebuild that trust.
“In short, prose specs can't be executed; tests can.”
9/10
Long-horizon agents and durable execution
Agents that run for days or weeks crash, retry and act on stale facts; wrapping the loop in durable execution and putting business rules in tool code lets them recover safely.
How it works
- Route every outside call (model decision, tool, message) through a logged step; after a crash, replay skips recorded steps and rebuilds state.
- Gate tools with an allowed(state) check, so rules like 'no order before approval' live in code, not the prompt.
- Use an idempotency key, such as the PO number, so at-least-once retries don't place duplicate orders.
- Re-check facts before irreversible actions, and have managers approve a price range with an expiry instead of one value.
- Version business rules in state so running tasks finish under the old rule while new ones use the new one.
Steal this: Before your next agent tool that does something hard to undo, add a deterministic allowed() precondition and a re-read of the underlying facts inside the tool, rather than trusting the prompt.
Go deeper if you're building agents that wait on people or external systems for days and need crash recovery, retries and safe deploys.
“This way the business rules that matter are in code, not in the prompt.”
9/10
I stopped babysitting my coding agents. Here's the workflow.
Running many parallel agents left the author as the bottleneck on decisions and bug-finding, so they switched to one owner agent that proves its work on the real app, checked by a different model family.
How it works
- Interview first: core-grill asks only about user-visible choices, then core-spec writes spec.md and verify.md, with checks written before any code.
- A different model family reviews both files read-only in a fresh session; then they are frozen and a draft PR is opened.
- One deliver agent owns the work, keeps a committed plan.md in ExecPlan format, and checks each milestone with a fresh subagent.
- A script, check-delivery.mjs, gates merge: frozen specs unchanged, report covers merged code, verdict PASS, verifier from another model family.
- Decisions and deviations sit at the top of the PR, each noting what to redo if overturned, so the human reviews decisions, not diffs.
Steal this: Before your next multi-step feature, write verify.md scenarios that start from a real user entry point and note what the old code does, so a check that could never fail gets caught.
Go deeper if you're the bottleneck on relaying decisions and catching late bugs in agent output, and want a complete, installable spec-verify-deliver loop with cited sources.
“When the agent struggles, we treat it as a signal: identify what is missing—tools, guardrails, documentation—and feed it back into the repository.”
9/10
GetMintOrg/Engineering: Guardrails for AI coding agents
Rules written in prose (CLAUDE.md) get broken eventually, so when you correct an agent twice on the same thing, turn the rule into a hook, lint rule or CI check that makes the mistake impossible.
How it works
- A PreToolUse hook blocks writes to .env files, keys and credentials before the agent touches them.
- A new-test hook requires a @guards tag naming the failure the test catches, and exit code 2 sends the reason back to the agent.
- Custom oxlint rules ban mocking your own modules, narration comments and hand-rolled sleep delays.
- A CI ratchet fails PRs that loosen lint budgets, grow exemption lists or add new disable directives.
- A ship skill takes a change to a green PR with every review thread handled.
Steal this: The next time you correct your agent twice for the same mistake, replace the CLAUDE.md line with a PreToolUse hook that exits 2 with a reason, like requiring a @guards comment on new test files.
Go deeper if you're repeating the same corrections to your coding agent and want ready-made hooks, lint rules and CI checks to copy.
“A rule in prose is a rule the agent will eventually break.”
9/10
Every PR gets an agent review before a human sees it
Reading about 50 PRs a week cold is slow, so each PR first goes to a read-only reviewer agent on a different vendor than the author, and a report reaches the human before the diff.
How it works
- Launch one scout agent per PR in its own pane; each can only read the PR and writes report.md locally.
- A binding charter forbids commenting, approving, pushing or merging, and requires findings ordered by severity with file:line citations.
- A fifth spec-review pass sees only the diff, commits and ticket, listing what was asked but not built, built but not asked for, built wrong.
- Pin the reviewer model explicitly, since role overrides skip scouts, and use a different vendor than the code-writing model.
- Cap review rounds at 3 (5 for security changes) and keep a ledger where every finding ends fixed, rejected with reason, or open.
Steal this: Before your next batch of PRs, write a read-only review charter (no posting, cited file:line findings, escalate auth and data changes) and run it on a different model than the one that wrote the code.
Go deeper if you're the bottleneck reviewing agent PRs and want the actual charter text, model pinning notes, and tested launcher scripts.
“The agent can read the PR and nothing else: it can't comment, approve, push or merge.”
9/10
Reflex: an architecture for building mobile apps with agents
Emulators make agent feedback slow, so Reflex runs app logic headlessly and exposes each screen's state and actions as JSON, letting a CLI and one shared test file drive both iOS and Android.
How it works
- Keep all non-UI logic in a core module with no UI imports, with platform services passed in through the constructor.
- Give every screen's ViewModel the same shape: actions in, state out, one update function that changes state.
- Add an introspection node per ViewModel exposing a name, state as JSON, and every action, with no conditionals.
- A CLI with ls and do commands reads and drives the focused screen, and the same tests run on both platforms.
- Block all network calls in tests and supply fixtures, so a suite of 1,700+ tests runs in about four minutes.
Steal this: Before your next agent-built feature, add a small JSON state-and-actions interface to one screen or service so the agent can inspect it and run a journey test in a terminal without a UI or device.
Go deeper if you're building mobile or UI-heavy apps with agents and want a fast, deterministic feedback loop plus one test suite keeping two platforms in parity.
“The code is written twice, the tests are written once.”
8/10
How we rebuilt the Shopify app in 12 weeks
Agents write code faster than humans can check it, so Shopify rebuilt its mobile architecture, tests and review flow so agents get fast feedback and humans only judge intent.
How it works
- Business logic runs without a device, so agents load a ViewModel from a CLI and check state in milliseconds.
- Architecture lives in examples, architecture.md files, lint rules and CI checks that fail when iOS and Android diverge.
- The legacy app serves as the spec; work splits into small checkpoints verified against it, not a written spec.
- An agent does an adversarial review against the guidelines first; humans then review small cross-platform PRs for intent.
- Each day an agent turns that day's review comments into updates to skills and Architecture.md, so one comment becomes a rule.
Steal this: At the end of each day, have an agent read all review comments from your PRs and fold the recurring ones into your skills or architecture.md file.
Go deeper if you're scaling agent output and finding that review, CI and merging, not code generation, are now your bottleneck.
“Agents need more than instructions. They need a way to discover when they are wrong.”
What the judge turned down this week (29)
- LLMs are terrible at designing AI systems
An interesting diagnosis of why agents choose symbolic or regex designs over LLM calls, but the page offers no concrete technique: the 'tell the agent to use a judge model' advice is the card's own inference, not something the post teaches or demonstrates.
- Your Algorithm: Giving People More Ways to Personalize Facebook Recommendations
The card accurately reflects the page, but it is a recommendation-system architecture overview, not agent harness craft: the layered LLM checks and the LLM-as-judge are described only at a high level, with no prompts, code, rubrics or evidence a reader could copy this week.
- Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents
This is mainly a research finding about weight-space backdoors wrapped in Foundry marketing. The only advice (verify provenance, diff weights, use least privilege, monitor) is generic security hygiene with no concrete audit procedure, and the page's promised starting-point list is empty, so readers get no specific harness technique to try.
- Fragments: November 10
This is a link roundup, like the calibration's Fowler Fragments fail: the CAFE(S) context checklist is only summarised secondhand without its context smells or examples, and the 70% verification figure is a statistic with no method behind it.
- Designing for Agents — Zeno Rocha, Resend
This podcast is mostly about agent-driven product discovery (AEO) and Resend's business; the only portable technique, expanding a frequently used MCP tool result and trimming what the model doesn't need, gets one paragraph, far short of the 8+ a podcast needs.
- What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team
A podcast about a data vendor's benchmarks and business: the page only names process-level grading, penalty traps and dependency gates in show notes and timestamps, without teaching how to build them, so it falls well short of the 8+ bar for a bonus listen.
- Agents aren't humans (but they kinda are)
A position piece on web identity and policy for hosted agents that also announces a coalition and Cloudflare features; it is not about shaping an agent's harness for verified output, and its concrete steps only work inside Cloudflare's products, so the card's 'steal this' overstates a proposal as a technique.
- Beyond code generation: A developer's guide to agentic engineering
Mostly a Copilot feature tour tied to GitHub Universe (Code Quality, Agent Merge, Dreaming, Fusion, AI Scan) whose portable ideas, stacked PRs and repo instructions, are named only at a high level with no code, examples or evidence, so it reads as vendor marketing like the Aviator and Google Cloud borderline fails.
- How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco
A podcast page with only a short blurb and timestamps; the card's mechanics (diff-understanding skills, agent-ready repos) are just topic headings, not a technique taught on the page, and nowhere near the 8+ a bonus listen needs.
- Intelligent Delivery: Making Every Unit of Compute Serve Better Ads
Mostly an ads-serving infrastructure post; the agentic part is one short paragraph saying agents read code, docs and experiment logs to shortlist changes with human approval, with no prompts, structure, files or loop details to copy.
- Cursor Projects is now multiplayer
This is a product launch announcement: shared context, the task board and agent-to-agent coordination are all features of Cursor Projects, with no portable mechanics or code, so the card's 'steal this' (a shared decisions doc) is the card's own extrapolation rather than something the page teaches.
- Shopify’s generative discovery model
A solid ML engineering post on training and serving a generative retrieval model (semantic IDs, RQ-KMeans, distillation, constrained decoding), but it is about building recommendation models, not about the harness around agents, so it is off-topic and has no agentic engineering technique to copy.
- How Workday Built an Enterprise-Scale AI Agent Platform with LangGraph and LangSmith
A vendor case study that reads as a testimonial: the only portable idea, wrapping your gateway, retrieval and auth in one internal SDK, is stated at a high level with no code, no design detail and no evidence beyond a time-to-prototype claim, and much of the benefit is credited to the product.
- The Jev Revolution: What Engineers Are Already Building with System One Models
A conference recap built largely around one product (Jev). The only technique-like idea, tournament-style chunk ranking, gets a single sentence with no code or evaluation, and Goyal's rules on when to use Jev aren't even on the page, so there's nothing concrete to copy this week.
- AI Engineer Code Summit: OpenAI on New Ways of Working With Codex
A conference recap of Codex product features (cloud handoff, voice as a control plane, dots) whose tips mostly only work inside OpenAI's products. It gives no files, configs or concrete harness mechanics to copy, so there's no transferable technique beyond 'hand off to the cloud' and 'use voice'.
- NaiveAI Promises 2,100 Tokens per Second. How Well Does It Code?
A model review with benchmarks and speed comparisons, which is an automatic fail; the only portable idea (compare medians and end-to-end time on fixed prompts) is a thin by-product of the evaluation, not a harness technique the page sets out to teach.
- The Data Product Platform Navigator
The page teaches a data mesh platform maturity model with a grid assessment, which is off-topic: it has nothing on agent harnesses, verification or agent output review, so it fails the agentic engineering topic despite a usable planning exercise.
- Libghostty 1.0
This is a library release announcement about C ABI stability (opaque handles, libabigail CI checks), not agent harness craft, so it falls outside the topic and is a launch post; the card mislabels it as harness design.
- Working with designers
An opinion essay on engineer–designer team dynamics and design systems, with no agent harness, verification or other agentic engineering technique to copy.
- A database is a store for global variables
A software design opinion essay about global variables versus databases, with no agent or harness technique to try; even the card admits it isn't about agent harnesses.
- Controlling Biology - Fabian Theis, Helmholtz Munich
A podcast on virtual cell biology with no agent-harness technique; the 'run a trivial baseline' takeaway is the card's own extrapolation from a passing mention of eval calibration, not something the page teaches for agents.
- How We Built TLX: A Hardware-Native GPU Compiler for Production ML at Scale
A GPU compiler post about warp-group orchestration in Triton, with no agentic-engineering technique: the ACE agent gets one sentence, and the card's 'steal this' (extend your existing tool instead of rewriting it) is generic advice, not a harness technique.
- On-Device Speech Recognition and Text-to-Speech in Google AI Edge Gallery
This is a product launch announcement for on-device ASR/TTS, not an agent technique. The few portable ideas are only mentioned in a single sentence, with no mechanics, code or evidence: stripping Markdown before TTS, sentence-level scheduling, and structured tags for code-switching. None of it is about agentic engineering.
- A2UI 1.0: Build delightful, fast, reliable, agentic experiences
A launch announcement for A2UI 1.0 with customer logos and roadmap. It has no code or walkthrough, and the only technique, restricting an agent to registered UI components, is a product feature rather than a harness practice taught here (similar to the failed A2UI & CopilotKit calibration item).
- Extending LLDB to Debug Applications That Span CPUs and GPUs
A solid debugger-architecture post about LLDB GPU plugins with nothing about AI agents, and the card's 'steal this' stretches a generic mock-backend testing idea into an agent technique the page never teaches.
- Announcing Long-Context Training on TPUs with MaxText Context Parallelism
A product launch post about TPU training infrastructure (context parallelism in MaxText). The techniques are real, but they concern model training systems, not the harness around an agent, so there is no agentic engineering technique here.
- From prompt to app: Build with Google Maps in AI Studio
This is a product launch announcement for Maps in AI Studio, and the only 'technique' (sandbox quotas) is a product feature rather than a transferable harness practice.
- What is Pathways? Single-controller training for large models
A GA launch post explaining Pathways' distributed training architecture for TPUs; it is about ML infrastructure, not agent harnesses, and the card's 'steal this' stretches it into an agent-loop technique the page doesn't teach.
- Glue work is still harmful in the age of AI
An opinion essay about glue work and career strategy with no agent harness technique; its only 'tip' (let an LLM draft write-only reports, write colleague messages yourself) is generic advice, not a transferable agentic engineering method.
This issue as markdown, as the harness committed it.