The check you didn’t know to ask for

Your agent can pass every test you wrote. Who notices the test you forgot? One customer books a workshop. The write succeeds, the reply disappears, and the screen says “try again”. The first booking already exists.

I want an agent that notices the awkward case before I spell it out. If the lesson transfers, the next project could benefit without someone having to make the same mistake first. The experiments below show both why that is worth testing and where the current machinery breaks.

Good answers need better questions. What this article offers is a runnable local example and a proposed learning loop. It does not establish expert-level AI review; discovery and human benefit are unmeasured.

A retry is not a second intention

The important question can be absent. A green checklist of UI, test and happy path says nothing about a concern it never included. Here is the one this example is built around.

1. First attempt: the booking exists, and the customer doesn’t know. The database has one row. A timeout says nothing about whether that write happened.
2. Retry: the same request creates a second booking. The naïve implementation repeats the effect. Your customer wanted one workshop, not two.
3. The fix: recognise the request and return the original booking. Use the same request identity within the same account. Check the row count after reopening storage.
4. The exception: another account’s booking must still work. A global duplicate key drops legitimate work. The fix has to preserve the account boundary, and a green test alone does not show that.

Two explanations fit the second row. One intent ran twice, or these are independent requests. To distinguish them, compare account, request identity and stored effect.

The starter executes isolated SQLite operations. It reopens the database, repeats a request, tries a new request, crosses accounts, and rejects a changed payload reusing an existing identity.

This is not an exactly-once guarantee for remote services, concurrent production workloads, payment effects or crash recovery. The supplied implementation and contract are authored examples, not independent model outputs. A hash identifies bytes; it does not authenticate truth.

{
  "candidate": "scoped",
  "checks": [
    {
      "expected": {
        "rows": 1,
        "same_id": true
      },
      "id": "retry_survives_reopen",
      "observed": {
        "rows": 1,
        "same_id": true
      }
    },
    {
      "expected": {
        "different_id": true,
        "rows": 2
      },
      "id": "new_intent_is_not_dropped",
      "observed": {
        "different_id": true,
        "rows": 2
      }
    },
    {
      "expected": {
        "different_id": true,
        "other_tenant_rows": 1,
        "rows": 2
      },
      "id": "tenant_scope",
      "observed": {
        "different_id": true,
        "other_tenant_rows": 1,
        "rows": 2
      }
    },
    {
      "expected": {
        "rejected": true,
        "rows": 1
      },
      "id": "conflicting_reuse_is_rejected",
      "observed": {
        "rejected": true,
        "rows": 1
      }
    }
  ],
  "contract_version": "lesson-1",
  "evidence_origin": "executed-local-sqlite-fixtures",
  "fingerprint": "96aa0eb3f5c3744c420281605ede5267b5595802d1d74920f2cb0b2401e21847",
  "production_release": false,
  "reason": "four local fixture invariants observed; not production approval",
  "scope": "shared-booking/local-sqlite",
  "status": "PASS_SCOPED_CHECKS"
}

Turn the hunch into something we can challenge

A hunch becomes useful when we can say what would change our mind. That means keeping the rival explanation and finding the discriminating observation.

The loop moves from meaning to observation to measurement. The meaning is “A retry shouldn’t book twice”, which raises its own questions: whose request, which operation, what legitimate exception? The observation is the same account, the same request, and a row count. Two rows support the concern; a different account changes it. The measurement asks whether the check helped on later cases, counting missed faults, false alarms, effort and regressions.

A person would put the distinction as “Retry this request. Don’t erase somebody else’s booking.” Context gives the rule its meaning. The retained lesson records it:

lessons/retry-one-intent.json

Concern

One intent, two stored bookings.

Exception

A different account is independent.

Observe

Reopen storage. Count rows and IDs.

Invalidate

The operation or scope changes.

The next agent runs python3 review.py --candidate scoped and gets four local checks passing. That is an inspectable result rather than “I think it works”.

That is the mixed qualitative–quantitative model. Context defines the question. Code tests explicit relationships. Statistics needs appropriate outcome data. A simulation can expose a bad assumption before a complete product exists; it cannot tell us how real users or models perform without observing them.

A prompt asks. A protected check can block. The host must require the approved check. The article cannot.

The possible payoff is useful judgement kept beyond this one review: fewer things only you remember to check, an agent that brings a question, a rival explanation and an observable test, and a repository of lessons that can change when the context does. The practical step is a retained check. Discovering new lessons is the research.

Another opinion is not another observation

Sixteen different states can display the same success message. Take four checks: durable save, one effect, correct account and usable recovery. Each either holds or fails, and all sixteen combinations can show a screen that says “Saved”.

The message alone has ruled out none of these possibilities. Each known-good observation halves what remains within the model: inspecting stored state leaves eight, counting stored effects leaves four, inspecting account scope leaves two, and inspecting recovery leaves one. Repeating the same Saved message leaves sixteen. Ask for evidence that separates the remaining explanations.

This is a noiseless finite teaching model. A remaining state is not proof that unmodelled faults, dishonest observations or human misunderstandings do not exist.

H(E) = { h in the declared model : h matches evidence E }
A claim is identified within this model only if
it has the same truth value in every h remaining in H(E).

Our learner knew the rule, not its missing context

Learning a rule from a supplied representation is not discovering a missing concept. In the retained v6 E601 counterexample, 0 of 64 answers are wrong in the original context. Flip the omitted context bit and every answer changes: 64 of 64 are wrong. The original-context control reuses training configurations. This is an adversarial finite example, not typical model performance.

A supplied scope guard defers all 64; a supplied correct relation handles them. Neither demonstrates autonomous discovery. Those are authored finite cases, not measured human or LLM error rates. The complete history and shared vocabulary remain in the archive. Auto-Rubric supplies related work on learning explicit criteria; it does not validate this workflow.

A working checker isn’t a discovery engine

Experienced reviewers are a comparator, not infallible ground truth. Compare ordinary strong-model review, the proposed process and human review using independently assessed outcomes.

The research unit is a cue, a contextual concern, a rival explanation, a discriminating observation and an outcome. The qualitative work defines the distinction. Mathematics makes its assumptions explicit. Statistics evaluates appropriate observations. Simulations test consequences before a complete product exists.

Test the full system against a strong ordinary agent and experienced reviewers. First hide the concern list. Then compare judgements on identical evidence. Finally test new cases after feedback. Measure useful discoveries, false alarms, regressions, human effort and completion, not just agreement.

A better judge cannot answer an unasked question

Start with 100 genuine issues in a hypothetical workload. Each probability is conditional on the previous stage, and evidence is obtained 90% of the time.

The chain is 100 × P(D | I) × P(O | D,I) × P(C | O,D,I), where I is a genuine issue, D that the concern is considered, O that adequate evidence is obtained and C that it is correctly confirmed. This is the chain rule, not an assumption of independent stages. Changing an input does not demonstrate that the corresponding improvement is achievable or free. Counts are expectations, so they can be fractional.

Spend on the unknown, reuse the known

A cheaper question is useful only if the whole review costs less. At the checked rate, 1,000 packed questions carry an estimated metered input fee of $0.0021. One needless five-minute review has an illustrative time value of $3.33 at US$40 an hour, though that is not cash spent. Token prices are only one term; spend effort on useful evidence.

• Discover (“What could this flow be missing?”): a strong model with tools.
• Interpret (“Does this message imply completion?”): a validated semantic judge.
• Verify (“How many rows survived the retry?”): a query or executable test.

Jev, an SLM, a classifier or a frontier model may fit interpretation. Compare the actual task, not model size alone (R1).

Send the context once instead of 1,000 times. In US dollars at the price checked on 20 September 2026, repeating the context costs $0.42168 and packing it costs $0.00210 for 1,000 questions:

(10,000 + 1,000 × 40) × $0.042 / 1,000,000
= $0.00210

That is 200.8× fewer billed input tokens, not better judgement. The assumed state is 10,000 tokens and the assumed complete question 40 tokens. The published Jev input rate is $0.042 per million tokens, and output is free. Its documented total and state-plus-longest-question limits are 64,000 and 32,000 tokens. Packing repeats state when a new batch is required. These estimates exclude real tokenisation, payload overhead, retries, observation and human work (R1).

No live Jev call occurred. The comparison is two input-packaging strategies at one provider’s rate, not Jev versus a frontier model, and not evidence that 1,000 worthwhile questions exist. A zero count means no semantic calls, not that code can solve any semantic problem.

The upkeep still has to earn its place

Keeping the check costs 55 minutes of setup and upkeep. In the starting case each reuse saves 4.85 minutes net, so time turns positive after 12 uses. Over 30 relevant future reviews, in this illustrative and unmeasured model:

The time model is Time saved = N[(r₀ − r₁) − f·t − (e₁ − e₀)·k] − F. N is uses, r is human review time, f is false flags per item, t is minutes per flag, e is escaped-fault frequency, k is rework time and F is setup and upkeep. The starting case uses 6 minutes of review before and 0.7 after, 0.15 false flags per item at 3 minutes each, e₀ = e₁ = 3% and k = 90 minutes. Categories must not double-count work.

Metered spend is separate from time value. The example uses $0.30 per original model review, and $0.0031 per replacement pass plus a 10% return to the original model. Those are invented workflow prices, not published frontier pricing. When the original model’s marginal charge is set to zero, the extra service costs more cash. Fixed subscriptions, quotas, hosting and harms beyond rework are not valued here.

A positive time result is not acceptance. In the faster scenario the escaped faults rise, so it is not a reliability improvement. Task mix, consequences and the real comparison need measurement.

A cheaper building block, not a closing window

These are dated sources, not a countdown, and not screenshots, endorsements or evidence that this workflow is faster.

• Anthropic, Demystifying evals for AI agents (January 2026). First-party engineering guidance: inspect outcomes, not just a convincing trace.
• OpenAI, Harness engineering (11 February 2026). A first-party engineering account of making project knowledge and constraints executable.
• TypeSafe model reference (checked 20 September 2026). $0.042 per million input tokens: bounded text judgements can be very cheap. That is a provider price, not validated performance.

Use your next costly correction as the test. That is the timing argument. I have no evidence this opportunity expires. If useful lessons transfer, a small team could keep more of its hard-earned judgement. A growing pile of noisy checks would do the opposite.

These sources describe available components and practices: tool-using agents, executable feedback and cheap bounded evaluation. They do not establish that our workflow improves productivity, learns human expertise or creates a time-limited commercial advantage. No market deadline is claimed.

Our work is strongest on authored, repeatable distinctions. The research challenge is discovering new ones from independent evidence, preserving their exceptions, and improving later outcomes. Novelty for rubric learning is not claimed. Model subscriptions and local constraints can overturn any small-model cost advantage.

A social post or a news screenshot would establish what was posted, not the claimed effect. The source-capture failure log is included in the package; no fake tweets, engagement counts, news mastheads or publisher screenshots were constructed.

If the lesson transfers, you don’t have to find it first

An agent checking a booking might learn to question premature success elsewhere: an upload, a queued job, a tool saying it finished. That could move some experienced lines of questioning into the workflow.

The condition matters: a familiar pattern is a hypothesis. Each new context needs evidence and its own exceptions. The current lab has not measured autonomous discovery, expert equivalence or that human payoff.

Give the agent a question worth keeping

1. Find one consequential concern the current tests miss.
2. Keep a plausible reason the behaviour is acceptable.
3. Run the observation that distinguishes those possibilities.
4. Save the scope, exception, check, and observed result.
5. Report unresolved questions. Don’t rewrite the acceptance contract.

Local demo; Python standard library; no API key.

No merge, deployment, spending or permission changes are authorised. What should come back is a fixture result, not a release certificate:

Changed

Account-scoped retry handling.

Observed

Four local checks; current source fingerprint.

Unresolved

Remote effects, production concurrency, unseen concerns.

# Keep the lesson, not just the patch

Use this on the next relevant coding task. Work in a branch or sandbox; do not deploy, merge, spend money, change permissions, or weaken existing checks.

1. Read the task and existing acceptance criteria. State the user outcome in one sentence.
2. Identify one consequential contextual concern that the current checks might miss. Use the actual task, not a generic list. Write a plausible reason the behavior could be acceptable. If there is no useful additional concern, say so.
3. Propose the cheapest observation that distinguishes those explanations. Run exact checks in code. Use a semantic model only where interpretation is needed; keep its version and uncertainty. Missing access means "not observed", not "passed".
4. On a confirmed recurring issue, propose a small lesson file: context, concern, exception, runnable check, expected observation and invalidation condition. Preserve valid alternatives. Never quietly rewrite the acceptance contract to make the implementation pass.
5. Run the agreed check against the current revision. Return the actual command, observed result and scope. A passing statement in your own prose is not a test result. Leave undiscovered mechanisms and user-comprehension claims explicitly untested.
6. End with: changed / observed / unresolved / time and metered usage. Ask for human review of a changed contract or a consequential unresolved decision.

For the supplied local booking example only:
`python3 review.py --candidate scoped --output receipt.json`

For a real repository, propose the smallest adaptation to its existing tests. Do not assume the demo checks verify that repository. Have the host make the approved check required and protect the checker/contract; this instruction alone cannot enforce anything.

None of them proved how the booking survived a retry

Imagine three reviews approving the button, the copy and the happy path. Each could be right about the thing it checked. None of them proved how the booking survived a retry.

I want to keep the missing question when I find it, and test whether the agent can learn the next one. Those three reviews are imagined. The research question is real; the full capability remains unproved.