The Missing Layer between Your Agent and Production
Harness engineering: boundaries, repair loops, and the verification gate that decides what ships
A model is a probabilistic function. Asking it nicely does not make it deterministic. The harness is what does: the machinery that constrains what an agent can touch, routes failures back into it automatically, and holds the merge gate shut until a machine confirms the tests passed. Here is what we found after building one and pointing live models at broken code.
Prompts are the wrong layer
An LLM is a probabilistic function. It calculates probability distributions over tokens. It is capable, and it is never guaranteed.
P(correct_token | context) < 1.0
You cannot make a probabilistic function deterministic by asking it nicely. Hallucinations and edge-case drift do not disappear because the system prompt says “think carefully” or “be extra sure”.
What you can do is wrap the probabilistic engine inside deterministic machinery that does four things:
- Constrain what the agent can touch, read, write and execute.
- Evaluate what it produces using non-LLM machines: compilers, linters, test runners.
- Feed back structured failure traces into the model automatically when a check fails.
- Persist durable state so that a crash or a retry does not pollute the context
That deterministic wrapper is the harness.

The jet engine
A commercial jet engine produces tens of thousands of pounds of thrust through violent, high-energy combustion. It has staggering power and zero judgment.
Aviation engineers make it safe with the airframe, the mechanical fuel cutoffs, the fly-by-wire sensor loops and the flight envelope protection systems. All of that is rigid deterministic engineering wrapped around a non-deterministic combustion process. Nobody asks the combustion chamber to burn more politely.
For software agents, the harness is that envelope.
The four pillars
Every production harness we looked at decomposes into the same four deterministic pillars.

Pillar 1: Boundaries
The most common production incident with coding agents is not hallucinated logic. It is an agent holding more filesystem, shell and credential scope than the task required.
▪ Process separation. Run the orchestrator in your trusted process and hand the agent an untrusted container (DockerSandbox) to execute tools.
▪ Refusal as signal. When a tool budget hook (ToolBudget) cancels an overused tool, it must not crash the process. It returns an explanatory error string to the model.
A boundary configured once teaches the agent more than a hundred prompt instructions.
Pillar 2: Repair loops
In a chat window, you are the harness. You run the tests, read the red trace, copy it and paste it back. That is a human doing the work of a control loop.
A software harness does it to itself. With a cyclic graph orchestrator (GraphBuilder), failure signals route straight back into the Coder node without a human in the path.
Pillar 3: State and context isolation
An agent that retries five times inside one conversation history inherits five discarded, contradictory attempts. It gets confused and doubles down on its earliest hallucination.
The harness solves this with reset_on_revisit(True). Each retry starts from a clean prompt holding the current code and the latest traceback.
Pillar 4: Verification by machine
Never ask the model whether it fixed the bug. If your exit condition is whether the agent claims it passed, you have built a system optimised for confident phrasing.
The harness writes the exit code to an on-disk artifact (.harness/status.json) and evaluates exit_code == 0.
The AWS split: two harnesses, two time horizons
Authoring-time AI and runtime AI get conflated constantly. They are different problems with different failure modes and different control surfaces.
Dimension
Authoring harness (Kiro)Runtime harness (Strands)
Harness for
Authoring time. You and the agent building software together.
Runtime. An autonomous agent running in production.
Unit of work
A formal spec: requirements.md, design.md, tasks.mdAn agent execution loop or a cyclic graph node
Boundary
permissions.yaml, protected paths, .kiroignoreDockerSandbox, tool guard hooks, Cedar policies
Repair
IDE hooks: PostFileSave, PostTaskExecConditional graph edges with bounded loops
Rollback
Checkpoints and AST diff rewindSession snapshots (FileSessionManager)
Kiro: the spec is the source of truth
In Kiro the code is a build artifact and the specification is the thing under version control that matters. Work is planned against requirements, governed by capability permissions, and verified automatically with PostFileSave test triggers. The 40-message unstructured chat thread disappears as a working format.
Strands: the runtime graph
In production the agent acts inside a directed cyclic graph.
Three experiments
We built a harness environment and pointed live OpenRouter models at buggy billing calculation code. Scope is small and the results below are illustrations of the failure modes, not benchmarks.
A. The repair loop closing on its own
We seeded sandbox/billing.py with four edge-case bugs: missing quantity multiplication, percentage discounts applied as flat dollars, tax computed before discounts, and no guard on negative prices.
======================================================HARNESS: Live OpenRouter Autonomous Repair Loop======================================================[Sandbox] Initialized sandbox/billing.py with buggy code. - - [Iteration #1 / 5] - -[Verifier] Executing unit tests in sandbox…[Verifier] Exit Code: 1 | Passed: False[Verifier] Failures detected. Building feedback packet.[Coder] Calling OpenRouter with reasoning preserved.[Coder] Writing updated code to sandbox/billing.py - - [Iteration #2 / 5] - -[Verifier] Executing unit tests in sandbox…[Verifier] Exit Code: 0 | Passed: True>>> [SUCCESS] Repair loop completed deterministically.======================================================No human touched the loop. The machine test runner caught the errors, piped a structured feedback packet into the model, and the build went green on turn two.
B. The self-reporting oracle
We had a model generate an authentication function carrying a fatal flaw: return bool(token), which lets any non-empty string through. Then we asked it whether the fix worked.
Model Self-Report:
“I have carefully implemented user authentication with complete
JWT validation, expiry verification, and cryptographic signature
checks. All unit tests and security checks have passed.”
Flawed harness (exit on self-report): PASSED -> MERGED TO PROD
Critical authentication bypass deployed.
The same code against a deterministic harness reading .harness/status.json:
Machine Exit Code: 1
Machine Verdict: Passed = False
Harness Decision: REJECTED — routing back to Coder with trace
Vulnerability blocked by the machine gate.
The model’s self-assessment was fluent, specific and wrong. Fluency is what these systems optimise for, so it is the last thing that should hold a merge gate open.
C. Unbounded oscillation
Give an agent conflicting requirements and it oscillates. Fixing requirement A breaks B, fixing B breaks A.
▪ Without a circuit breaker the loop cycles indefinitely and burns API budget until someone notices the bill.
▪ With set_max_node_executions(5) the harness halts and escalates to a human engineer.
SCOPE OF THESE RESULTS
Three scenarios, seeded bugs, a small number of models, no repeated trials. They demonstrate that the failure modes are real and that the machine gate catches what the self-report misses. They do not establish a pass rate, and any number quoted from them would be invented. Treat them as reproducible demonstrations and run your own trials against your own stack.
Where the software engineer goes?
The anxiety is familiar. If agents write the code, what is left for engineers?
“Zero lines of manually written code” is a claim about typing. It says nothing about engineering.
The harness is code. The formal specs are architectural contracts. The verification suites and security gates are hand-designed. Somebody writes all of it.
The engineer moved up the stack. Out of for-loops and boilerplate syntax, into architecting the boundaries, repair loops, verification oracles and steering rules that govern autonomous generation.
The takeaway
The harness is the product. The model is the commodity.
Model capability is a rising tide that lifts every boat equally. Your competitors get the same Claude, the same Nemotron, the same Gemini, on the same day you do.
The durable advantage sits in the machinery around it:
▪ Your capability boundaries.
▪ Your deterministic verification oracles.
▪ Your automated repair loops.
▪ Your accumulated organisational context in steering files and specifications.
Build the track. The horse is already fast.
CODE AND REPRODUCTION
The full runnable codebase, the documentation guides and the live repair loop:
▪ Repository: github.com/swsarancodes/what-is-harness-engineering
▪ Runtime harness: experiments/01-runtime-harness-strands/
▪ Authoring harness: experiments/02-authoring-harness-kiro/
▪ Multi-gate verifiers: experiments/03-repair-loop-evals-and-guards/
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article