Keep the
big model.
Change its job.

Keep the model that helps you understand the requirement, research options, write code and diagnose failures. Distinguish that build-time agent from the components called repeatedly inside the shipped application. They can be the same model, but need not be.

This is not a claim that frontier models cannot use tools, return structured data or do arithmetic. A capable baseline retains its normal tools. The question is whether a different output representation, a tested specialist or deterministic implementation improves the entire feature.

For this example, segmentation produces editable pixels; a writer produces a draft; supplied seller information establishes condition; fee code applies a declared rule. A 3D reconstruction is an optional preview. These are proposed boundaries, not a tested marketplace architecture.

Both receive the same assumed token count, not necessarily the same amount of text: tokenisers differ. Small calls are charged even when they escalate. The cache option omits cache-write and warm-up costs, so it is a steady-state sensitivity case, not a complete cache bill. [pricing]

Images, geometry generation, taxes, tool calls and errors are not separately priced. Add incremental costs to overhead or replace the model with measured itemised costs. Your coding subscription is a different expense from the runtime API bill. A subsidised subscription is not unlimited free deployment compute.

Do not optimise this number alone. Compare cost per accepted outcome only when the task population, acceptance criteria and complete costs are available. Keep severe failures separate from an average score. Try caching, batching and simpler code before introducing another service.

A helpful guide can also disrupt work the agent already did correctly. Let p be baseline success, r the share of failures repaired, and h the share of successes broken. These are conditional rates.

The guide, the helpers and their combination are separate interventions. Compare them with the existing capable agent. Use pilot tasks to choose one package, then freeze it and confirm on fresh task families. Preserve baseline successes and near-duplicate lineage; repeated attempts are not new independent tasks.

SkillsBench reports improvements on selected tasks; its construction rejects tasks without measurable separation. An AGENTS.md study found increased cost without general success improvement in its settings. Neither establishes the effect of this article or kit. [research]

Test the obligation, not the confidence.

Verification: do code, constraints and calculations satisfy the declared contract? Use unit, property, integration and numerical checks, with independent oracles where feasible. Validation: does the finished feature meet the seller’s actual needs? Inspect corrections, missing items, usefulness and the work displaced to people.

For a mask, inspect ground-truth coverage and edge errors. For a decision model, examine false acceptance, calibration and abstention under realistic shift. For a small writer, count invented claims and edits. For geometry, inspect topology, views and dimensions against suitable reference data. No single score covers all these contracts.

Numbers plus failure explanations.

Record all planned tasks, latency, provider charges, operations and human review. Qualitatively code wrong assumptions, unnecessary steps, hidden edits, requirement mismatch, severe failures and unknowns. A second model’s agreement is not independent truth.

Declare the primary comparison, useful-effect threshold, risk boundaries, budget, independent unit and stopping rule before acceptance data. Use grouped inference for repeated task families. Report uncertainty; an inconclusive pilot is not proof of equivalence. More peeking can manufacture apparent gains.

The archived lab’s prospective calculation found 27.2% power for one assumed five-point improvement with 120 independent pairs and 20% discordance under a fixed one-sided test. That is a chance of detecting an assumed effect, not the chance this kit works. Its model, assumptions and raw results are preserved in the original archive. [lab]

Stop when the next test is not worth its cost.

Test the smallest consequential uncertainty first. Expand after a result survives its checks. Quarantine stale lessons, remove redundant modules, and revalidate when models, tools or requirements change. The intended result is an accumulating evidence base—not an ever-growing testing ritual.