Fieldnotes / engineering

Loop engineering

Explanation

Act, observe, decide

An agent loop alternates an action, an observation and a decision about what to do next. The controller enforces continuation and stopping rules.

A loop requires evidence and boundaries; repeated prompting alone does not enforce them.

  1. 0s The agent proposes an action. The tool executes the permitted operation.
  2. 7s The tool returns an observation. The controller receives actual evidence.
  3. 14s The controller checks the stopping rules. It decides whether another step is allowed.
  4. 21s If allowed, another action can follow. The cycle has an explicit decision point.

Use a specific failure

A discount-calculation test fails with a concrete assertion. The agent can connect that observation to a targeted code change.

A failing assertion is useful only when the relevant tests actually ran. Preserve the failing input and expected behavior.

  1. 0s Twelve tests run against the code. One assertion fails.
  2. 7s Send the failing assertion to the agent. It shows expected and actual behavior.
  3. 14s The agent makes a targeted repair. The code changes; the evidence is now old.
  4. 21s Return the changed code to the runner. The next observation must be fresh.

Stop after verified success

The repaired code runs through the selected suite again. When the configured success criteria are met, the edit loop stops and hands over evidence.

Passing this suite supports the tested behavior. It does not establish that all possible behavior is correct.

  1. 0s Run the same relevant checks again. This time, test the repaired revision.
  2. 7s All twelve selected tests pass. The runner reports what actually ran.
  3. 14s The controller checks the success rule. The observed result satisfies this rule.
  4. 21s Stop editing and hand over the evidence. Unrequested edits add uncertainty.

No tests is not a pass

A mistyped test selector can select zero tests. Under this repair-only policy, the agent stops rather than treating the empty run as proof of correctness.

Distinguish assertion failure, collection errors and no tests selected. They require different next actions.

  1. 0s A selector matches no tests. No behavior has been checked.
  2. 7s The runner returns that empty selection. The agent cannot call it a passing suite.
  3. 14s This policy stops further code edits. Keep the command and selection output.
  4. 21s Hand the evidence to an engineer. Repair the test setup before continuing.

Limit repair attempts

A controller can cap time, tool calls or repair attempts. Repeated unproductive failures should lead to a useful handoff rather than an endless loop.

The controller must enforce the budget outside the model. A handoff should explain what was tried and what still fails.

  1. 0s The task has a configured attempt budget. The controller counts completed attempts.
  2. 7s Another repair attempt still fails. The remaining allowance decreases.
  3. 14s The budget is exhausted. The controller blocks another edit cycle.
  4. 21s Send the remaining failure and evidence. A human can choose a new approach.
Read the guide

Helpful first: Harness engineering.

The idea

Here, loop engineering means designing the repeated action, observation, and decision cycle around a model. The harness—the software running the agent and its tools—must enforce boundaries. “Keep fixing it” leaves success, permitted actions, and stopping ambiguous; a persistent prompt does not enforce permissions.

Where it helps

Delegating the keyboard delegates decisions engineers previously made themselves. Running no selected tests needs a different response from a failing assertion. Anthropic’s engineering guidance supports environment feedback, stopping conditions, and human checkpoints; it does not establish an optimal attempt count or promise convergence.

Teams adopting coding agents need to make their existing judgment about feedback and stopping explicit before delegating repeated edits. Practising a small repair loop gives the team an inspectable contract they can reuse across agent tools.

An agent repairs a discount calculation. Task A runs 12 tests: one fails, a patch is made, then all pass. Task B uses a mistyped keyword selector and selects none (all discovered tests are deselected). Under this team’s repair-only policy, task B stops edits and sends the command and logs to an engineer.

What you will be able to check

  • Classify a test result as usable failure, usable pass, or blocked validation.
  • Implement and demonstrate stop conditions that the model cannot override.
  • Produce a handoff tying the patch, command, results, and stop reason together.

What changes in the team

Today
An engineer reads a failing test, edits the discount function, reruns the selected tests, and decides whether the output warrants another edit.
With an agent
An agent proposes scoped edits while a harness runs the agreed command, records its evidence, and routes the result through enforced continuation rules.
Still your responsibility
The engineer owns expected behaviour, test adequacy, edit scope, budget selection, and the decision to accept the patch or investigate a blocker.

Work through an example

  1. Agree the contract. Work in a disposable checkout, select the discount tests, and confirm the intended cases are collected. Allow edits only to the calculation module. For this exercise, allow at most three patches, six test invocations, and five minutes overall; give each command a timeout. These are teaching values, not measured production defaults.
  2. Record observations. Save each command, working directory, patch identifier, selected and executed test counts, exit code, and output. pytest documents exit 0 for success, 1 for test failures, and 5 for no tests collected. Use the runner’s report too: a numeric exit code alone cannot show that the intended checks actually ran. An unmatched -k selector can collect tests but deselect every one, also producing exit 5; distinguish collected, selected, executed, and skipped counts.
  3. Route the outcome. For a reproduced assertion failure within scope and budget, provide the failure details for another patch. For the agreed tests passing on the current patch, stop and request review. For absent tests, command errors, or unavailable dependencies, preserve the evidence and stop this repair-only run; diagnostic automation would need its own explicit scope.
  4. Evaluate the loop itself. Replay representative successful repairs and blocked runs. Check whether stops prevented wasted edits and whether handoffs let someone resume without guessing. Repeated identical evidence is a reason to investigate, not proof of progress; choose any no-progress threshold from observed team tasks. Compare failure identities and relevant assertion details under the same test selection; equal failure counts can hide entirely different failures.

The engineering decision

Enforce continuation in control logic. Require evidence for the current patch, preserve the test selection, and bound tool calls and elapsed time as well as edits. A passing selected suite permits review, not automatic merge. Three attempts is an illustrative policy to evaluate, not a universal recommendation.

  • Treating a missing test selection or silently skipped coverage as a successful repair.
  • Accepting yesterday’s green result after another edit, or letting the agent weaken assertions to get green.
  • Rerunning a flaky test until it passes without investigating the uncontrolled state described in pytest’s guidance.
  • Retrying a timed-out publishing tool blindly: Amazon documents that failure does not prove side effects were absent.

Try it with your team

Make the repair loop refuse missing evidence

  1. Use a small disposable function and a known failing test; record the exact intended selection.
  2. Wrap the test command with the three outcome routes and the exercise limits above.
  3. Run a repair, then run a separate task with a keyword selector that matches no tests.
  4. Compare the traces and inspect the patch and blocker handoffs with a teammate.
  5. Force a budget to expire and verify that another edit or command is refused.

You have an answer when…

  • The valid failure can lead to a patch and review; the no-tests run makes no product edit.
  • Each trace includes the selection, current patch, command, outcome, and stop reason; forced limit exhaustion also stops execution.

Read further

Choose one repair workflow: what evidence permits another edit, what ends it successfully, and what must be included when the run stops without a fix?

Original video comparisons