We ran the fifth loop for real. This is what changed

A field report on the optimization loop from our last post: five iterations of an agent editing our development playbook, with the receipts, the one that backfired, and the two changes we still refuse to call wins.

In our last post we laid out four loop types for agentic coding and then argued for a fifth: a loop that improves the process your agents follow to build the code, rather than the code itself. A builder delivers real work, a scorer grades it against a fixed bar, and an optimizer makes one bounded change to the playbook, with a falsifiable prediction attached.

Concepts are cheap. So we built it. This is the report from running it on one of our own repos, including the parts that went sideways.

The setup

The loop lives in a single Claude Code custom command, /iterate, pointed at one of our internal product repos. It optimizes the repo's process guidance: the CLAUDE.md where we define our dev workflow, the steps every feature goes through (plan, TDD, multi-agent review, merge gate), plus a few path-scoped rule files. One thing it can never optimize: the quality bar. The Definition of Done and the safety sections are frozen, and the optimizer is hard-blocked from touching them.

Each iteration dispatches three subagents in strict sequence:

1. A builder delivers a real ticket by following the current playbook step by step, and logs telemetry as it goes: how many agent dispatches, how much rework, where it stalled, and a list of concrete waste items, each pointing at the exact process file and step that caused it.
2. A scorer runs build, lint, tests, typecheck, and render against the frozen Definition of Done and reports a score. This is the guardrail. A faster process that produces worse software fails here.
3. An optimizer first adjudicates the previous iteration's prediction (and reverts the previous change if the prediction was falsified or the score dropped), then picks the single highest-leverage waste item and makes one edit to the file it points at, logging a new prediction that the next iteration can prove wrong.

Everything runs on a dedicated branch. Nothing reaches main without a human at the merge gate. And the last iteration makes no change at all, only adjudication, because a change made on the final lap can never be tested. A cap of five iterations buys you four tested changes, and you size the cap knowing that.

This is Anthropic's evaluator-optimizer pattern pointed one level up: the artifact being generated and graded is the playbook itself.

The first run taught us the measurement can lie

Our first run moved the overall workflow score from 92 to 96. It also handed us a lesson we nearly missed. One iteration appeared to regress, and the honest explanation turned out to be embarrassing: iteration one had opened a pull request and later iterations hadn't, so a chunk of the score delta was an artifact of the harness, not the process. The loop was grading two iterations that weren't run under the same conditions.

So before the second run we changed the loop, not the playbook. We added a harness-parity rule: every iteration runs under identical conditions, same PR handling, same artifact retention, same environment, and any check that the harness makes unscoreable gets marked N/A in every iteration. If you take one thing from this section, take that. Before you trust a loop to optimize a process, make sure a score delta can only mean what you think it means.

The final run, by the numbers

The re-run delivered the same two tickets every iteration (a backend row-level-security isolation guard and a frontend dialog discard guard; a moving target would have confounded the telemetry) across five iterations on one branch.

• Application quality: 96 on the first iteration, 100 on the second, then 100 every iteration after. Roughly 2,100 backend tests, zero failures, every run.
• Build time for the same work: 38 minutes in iteration one, then 13, 23, 12.7, and 19.7.
• Agent dispatches: 4, then 3, then a spike to 6, then 3 and 3.
• Playbook edits: four, one per iteration. Two ended the run CONFIRMED. Two ended it UNTESTED.

That last line is the part we care most about.

The change that backfired into the best win

Iteration three is where the interesting failure happened. A code reviewer flagged a finding as "Important": consolidate two database calls into one. Correct observation, except the ticket's plan document had deferred that exact change, in writing, in an "out of scope" section. The playbook gave reviewers no instruction to read that section, so the finding triggered a full extra review round. Agent dispatches doubled from 3 to 6 in that iteration, all of it spent re-litigating a decision that had already been made.

Iteration four's telemetry then surfaced something subtler. The false positive didn't recur, but only because the orchestrating agent remembered the previous iteration's postmortem and manually pasted a warning into the review briefs. The builder's own notes flagged it: the fix existed in one agent's memory, not in the process, and memory doesn't survive to the next run.

So the optimizer made iteration four's change: a paragraph in the review step telling reviewers to read the plan's scope section first, and to tag any finding that matches an explicitly deferred item as deferred-by-plan (confirm still acceptable) instead of Critical or Important, with a default of "unsure? raise it, don't downgrade." The attached prediction was specific: next iteration, findings matching deferred items get that tag, citing the process text rather than anyone's memory, review stays single-pass, and no genuinely in-scope defect gets downgraded.

Iteration five checked every clause. Three reviewer findings matched plan-deferred items, all three got the exact tag with the exact plan sections cited, review completed in a single pass, and six findings not covered by the plan went through normal severity untouched. Score held at 100. Confirmed, kept.

That one change is the whole argument for this kind of loop in miniature. The waste was real (it had a measured cost: three extra dispatches), the diagnosis came from telemetry rather than vibes, the fix was one bounded edit, and the proof was a prediction someone wrote down before the evidence came in.

The two changes we refuse to call wins

This is the discipline part, the part we'd expect most implementations to get wrong.

Iteration one's change was a carve-out letting the orchestrator skip a second reviewer dispatch when the only fixes were cosmetic, comments and formatting, no logic. In iteration two, dispatches did drop from 4 to 3, exactly the headline the prediction promised. A lazier loop declares victory right there. Ours didn't, because the telemetry showed why dispatches dropped: the review pass came back clean on the first try, so the carve-out never fired. The metric moved for a reason that had nothing to do with the change. Verdict: UNTESTED. The change stays, it's harmless, but it is not a win and the log says so in bold.

Iteration three's change (skip per-stage ticket bookkeeping for small tickets) hit the same fate for a different reason: the precondition never occurred, in any iteration, because the test ticket happened to already have its bookkeeping in place from a prior run. Untested, kept, not claimable.

Two confirmed, two untested, zero falsified, and the untested ones labeled loudly enough that nobody, including us six months from now, can mistake them for proven. If your optimization loop can't produce the verdict "this change did nothing we can measure yet," that's marketing.

The same loop, pointed anywhere a playbook lives

The dev workflow was one instance. The pattern needs exactly three things: work that follows a written process, a quality bar you can hold fixed, and telemetry on where the process wastes effort. That describes a lot of things that aren't software. Three we'd build:

Take a sales outreach playbook. A builder agent researches each prospect and drafts the sequence the playbook prescribes, logging every place the playbook fought back, from research steps that turned up nothing usable to a draft held up by a compliance question the document never answered. A separate reviewer checks every claim for accuracy and the send list for consent, and that check is the one thing bolted shut. Let a loop relax its own definition of a true claim and it will get faster at writing false ones. Everything upstream of the check is fair game for the optimizer: research order, templates, follow-up timing, one edit per cycle.

Support runbooks work the same way. Resolving real tickets by the book surfaces the waste on its own: macros that need rewriting before they can be sent, escalation criteria that take a judgment call, steps that make the customer repeat information. Reopen rate and escalation correctness form the quality bar, scored by an evaluator the optimizer can't edit, and the identity-verification and refund-approval gates are off-limits forever. A support loop that can trade a security check for a faster close is a breach with good response times.

Incident response is the one where the frozen part matters most. Severity definitions and rollback approvals never move, for the same reason our optimizer can't touch the Definition of Done: the loop may improve the path to the gate, never the gate itself. Around that fixed core you don't even need to wait for production to burn. Run a game-day drill against the current runbook, grade it on a fixed post-incident rubric (time to mitigation, communication completeness, rollback correctness), and let the optimizer change one runbook step per drill, with a prediction about the next drill's delta.

In each case the roles are identical: builder executes the document, scorer holds the fixed bar, optimizer makes one falsifiable edit, human approves what merges. Only the document changes.

What we'd tell you to steal

The constraints are the part worth stealing. Every piece of value in this run came from a rule that made cheating impossible:

• One change per iteration. Two changes at once means neither can be adjudicated.
• Every change ships with a prediction written before the evidence exists, and the next iteration's first job is to check it.
• Three verdicts: CONFIRMED, FALSIFIED, and UNTESTED, the one that saved our integrity twice. A metric that moved for the wrong reason proves nothing.
• Falsified or regressed means reverted. No defending a bad idea.
• The scorer's bar is frozen and the optimizer physically cannot edit it.
• The last iteration only adjudicates. A cap of K gives you K minus 1 tested changes.
• A human sits at the merge gate. The loop proposes on a branch; it ships nothing.

And one caveat we log against our own results: these four edits are validated for this repo and this workload. Your numbers will differ. What a loop finds and how fast it converges depends on the process it's pointed at, the system it runs on, the person who set the quality bar, and the model doing the work. Before any of these edits gets promoted to another repo, it owes us one confirmation run on a different ticket in a different codebase. A process improvement that only works on the ticket it was tuned on is overfitting with a changelog.

The first run took the score from 92 to 96. The final run took it from 96 to 100 and held it there, while the same work ran roughly twice as fast. But the number we trust most from this whole exercise is smaller: two. Two changes, proven, with the prediction and the adjudication in writing. That's what the fifth loop buys you: a process that can show its work.