The bug shouldn’t need you twice

Imagine the payment notification for that purchase is delivered a second time, and the handler grants on every delivery. Someone has to notice, work out the required behaviour and explain the fix. Without a check that captures it, the next builder can need the same explanation again.

The valuable part of this fix is the behaviour you establish. Keep a check the next builder can run, instead of explaining the same failure again.

This is Part 2 of a field note on agent work and human review. The examples are authored programs that run locally; nothing touches Stripe or a real account.

The demo hid the failure

Every implementation passes the first delivery. The failure shows up only when the same purchase is delivered again, which is exactly the case a quick demo never exercises.

A different message can still be the same purchase

Deduplicating event IDs is a useful first thought. But this business promise is about granting a purchase once, and an event ID is not necessarily the purchase being fulfilled. Three authored implementations show the difference:

• Every delivery treats another notification as another grant.
• Remember the event stops an identical replay, but not a second event for the same purchase.
• Remember the purchase keeps the account and purchase identity together.

This is the question worth keeping: which identity names the thing we promised the customer? A fast model can patch the wrong identity very efficiently. The expectation needs to survive the model change.

BEGIN TRANSACTION
claim (account_id, purchase_id) uniquely
if claimed: record the credit grant
if duplicate: return the existing outcome
if conflict: stop and inspect
COMMIT

The next builder should inherit the hard part

You already paid attention once to work out the required behaviour. Keep the input, expectation and result together. That is what can make a later repair cheaper.

1. Save the case, so the mistake can happen again. Keep the same-purchase input sequence, evt_01/buy_42 then evt_02/buy_42, and the expected 100-credit result: different events, one entitlement. A screenshot of a green first purchase cannot express this requirement.
2. Fix the implementation and leave the expectation alone. Let the agent repair the candidate while the acceptance expectation stays independent. The purchase-keyed function returns 100 for the same case, and the expectation has not changed. If the expectation is wrong, that is a separate decision rather than a quiet test edit.
3. Check the other direction: a new purchase must still work. Rejecting every notification can stop duplicate grants. It also stops valid customers receiving credits. buy_43 is a new purchase, so the balance must become 200; rejecting everything is not a fix. Keep the positive case and the exception together.
4. Hand it on, then compare another builder. Another builder inherits the same cases, and another model can attempt the same task with the same check. Charge the fallback, repeated runs and your review time before calling the cheaper model cheaper.

acceptance / purchase-grant.test

one purchase + repeat notification
→ expected balance: 100

another purchase
→ expected balance: 200

not: every event grants
not: block every new purchase

A check is a maintained asset only while its contract remains right. This starter cannot prevent an agent with write access from rewriting its own judge. Keep the result you care about independent of whoever writes the next version.

The cheaper model might still cost more

The goal is less repeated discovery, not free checking. The test makes outcomes comparable; it does not guarantee savings. Use the same hypothetical batch model as Part 1 and count the work that comes back to you.

A runnable failure is not a universal verifier

A useful check can be incomplete. Extra instructions can also make an agent do more work without improving the result (Gloaguen et al., Evaluating AGENTS.md). Keep the contract small enough to inspect.

The opportunity is to move repeated explanation and rediscovery into a check, give another builder a concrete result to reach, and spend your attention on a changed requirement, an important trade-off or a failure the system cannot resolve.

The countercase matters as much. If the task rarely repeats, the check is expensive to maintain, or the agent can optimise around a weak test, this can add process without value. Keep no-improvement runs in the evaluation.

Recent agent-engineering work makes reusable feedback worth testing. It does not make regression testing new, or this kit uniquely capable. The practical moment is the next bug you already understand, before you move on and the context disappears.

Keep the check, and keep its scope with it

Three invented handoff messages, “Implemented.”, “Looks good.” and “Works in the demo.”, say nothing about how the handler behaved on a retry. The check now shows a particular failure and a particular repair. Keep its scope with it. The next model gets to run it too.

Part 1, Your agents gave you homework, puts the human review bill back in the picture.

For coding agents: give the next builder the failure

The useful handoff is the input, the expected result and the command that distinguishes a fix from another confident answer. The instruction below is editable and needs no account or sign-up.

Nothing is sent or collected. Edit this before reusing it.

The kit is local teaching code plus instructions. It does not install a host skill, evaluate arbitrary code or protect a check the builder can rewrite.