Harness tuning is folklore
The argument is over. The harness matters more than the model, and there are now numbers on it. Princeton’s CORE-Bench moved from 42% to 78% on the same model with a different scaffold. Vercel cut a coding agent’s tool count by 80% and watched success go from 80% to 100% while a run dropped from 724 seconds to 141. LangChain took Terminal-Bench 2.0 from 52.8 to 66.5 without touching the weights. MongoDB’s Bazeley, Kumar and Xu collect these in a post on the agent harness, and they are fair enough to include the other side: Scale AI’s SWE-Atlas and METR both found harness choice inside the margin of error for some model families.
Winning that argument buys you nothing on Monday. “Change the harness” is not an instruction. Someone trims four words from a tool description and the agent stops reaching for it at the wrong moment. The retry cap goes from five to two and the weekly bill halves. Both changes are real and neither is explicable afterwards, because they shipped in the same week as a prompt rewrite and a model point-release. So you keep them and carry them forward as lore.
Why the edits never add up
Lin et al. put the problem precisely in Agentic Harness Engineering: a heterogeneous action space across editable components, trajectories voluminous enough to bury the signal, and edits whose effect is hard to attribute. Only the first of those is about the harness. The other two are about what you kept.
Observability usually gets scheduled for after the agent works. It is the instrument that tells you whether it does.
What a run should leave behind
Zhong and Zhu grade harnesses by exactly that in AI Harness Engineering. Their ladder runs H0 to H3, and at the top a run emits an auditable episode package: a reproduction log, a failure attribution, a verification report. The question shifts from whether the model produced a patch to whether the system can show its work.
Once that evidence exists, tuning becomes a loop you can close, and AHE closes it. Every harness edit ships with a written prediction of what it will fix, checked against the next round’s outcomes, which makes each edit falsifiable rather than merely plausible. Ten iterations took Terminal-Bench 2 pass@1 from 69.7% to 77.0%, past the hand-designed Codex-CLI harness at 71.9%, and the gains carried across model families by 5 to 10 points.
The automation is the headline. The falsifiable contract is the part worth stealing even if you never automate anything: write down what you expect the edit to fix, before you make it. That one habit would have settled most of the harness arguments I have sat in.
The same move, aimed at you
Zhou et al. call the underlying pattern externalization. Memory externalizes state across time, skills externalize procedure, protocols externalize interaction structure. Capabilities the model was once expected to recover on its own now live in the runtime around it.
Observability is that move aimed at the engineer. Why the search tool returns excerpts, why the retry caps at two, which of last quarter’s changes was the one that helped: that is knowledge you hold in your head today and lose when you change jobs. Traces put it somewhere the next person can read. Or the next agent. MongoDB’s post arrives at the same place one level up and calls it the platform layer, with durable execution, identity propagation, cost attribution and audit evidence sitting under the harness. None of those four is really an infrastructure problem. They are all questions about what your system can prove after the fact.
A harness you cannot instrument is one you will tune by anecdote for as long as you own it. The traces would settle it, and most teams delete them when the run ends.