How to let an AI use your browser without watching every click

A demo on my phone showed a browser agent finding a flight from Zürich to London in 7.1 seconds for less than half a cent. My first thought was obvious: use this for the browser work I keep putting off, such as university portals, finance dashboards, job sites and services with no useful API.

The demo answered the clicking question. It did not answer the trust question. I started with a fast browser demo and ended up redesigning the whole control loop.

I am testing the design on one read-only university workflow before I let it spread to anything more consequential. The companion article, The browser is becoming the API for every app, explains why the browser may become the API for every app, and why verified workflow evidence is the durable advantage. This one is the implementation.

Split the run into six responsibilities

The system became easier to reason about once I stopped treating “the agent” as one thing. Give each uncertainty to the least powerful controller that can handle it, then require independent evidence before success.

1. Outcome contract. What state must exist? Which sources count? What is prohibited? The objective is frozen before the run starts.
2. Frontier planner. Handles novelty and chooses a typed macro-option.
3. Capability gate. Compiles the legal action set, and deterministic policy checks authority, scope and budget. The model cannot widen it.
4. Jev or specialist. Makes bounded local choices from observed options.
5. Deterministic executor. Acts idempotently and preserves unknown outcomes.
6. Independent verifier. Reads back reality and issues a receipt, so the run can finish without the model grading itself.

When a step fails, the loop goes to recovery, which means re-observing or stopping, instead of declaring success. Nothing moves until authority and evidence exist.

In my setup, OpenClaw schedules and coordinates the run. Jev chooses among typed local actions. Software owns permission, state and side effects. Evidence decides whether the job actually finished.

The example: check UON at 6:00 and never submit anything

This is the bounded outcome used throughout the article. It reads Canvas, MyHub and UON Outlook, returns changed obligations plus source evidence, and never submits, messages, enrols, accepts or pays. It is specific enough to test and consequential enough to expose weak assumptions.

A click is not a result

My first design had a familiar agent loop: observe the page, choose an action, act, repeat. It looked sensible in a diagram. It was too weak in practice.

The problem was the unit of success. I was measuring whether the agent completed a sequence of actions. I needed to measure whether the external state I cared about now existed.

I also reused an idempotency key too broadly. A new scheduled observation could return an old receipt. The key now identifies one logical run, not one snapshot forever.

These were contract failures. A larger model would have made the same weak assumptions more fluently.

Use the least powerful controller that can reliably do the job

Different kinds of uncertainty need different controllers. Four tasks from the same workflow show where each one belongs.

frontier plan → typed option → policy mask → Jev choice → tool → evidence

Jev is useful when the choice is fuzzy, local and bounded

TypeSafe's Jev evaluates typed questions against structured state and returns typed answers with probability distributions. In a browser loop, that can mean choosing an operation and one compatible element from the page that was actually observed.

That is a smaller and safer problem than asking a model to invent JavaScript, selectors, destinations and recovery logic. In the reference package's demo, the observed page has four controls: #12 Past courses, #17 Current courses, #24 Help and #31 Profile. The target is #17.

The decision still escalates. The top choice is 82%, the local pair-error upper bound is 1.8% and the action budget is 1.0%. The typed choice is useful, but it has not earned autonomous execution.

The Jev Ultrafast repository treats DONE as loop termination, not proof. It also warns against blind mutation retries. Jev cannot grant authority, retrieve an unrestricted vault, or establish whether the real-world outcome happened. Those jobs remain outside the model.

Small per-step error becomes a large system problem

If a workflow has independent steps with the same success rate, the chance that every step succeeds is pⁿ. That is a useful warning even though it is not a production model.

Asking the model to be more careful does not fix this. Reduce probabilistic steps, checkpoint meaningful stages, store durable state, and verify postconditions.

A trustworthy action leaves a chain you can reconstruct

Operational logs are not enough. I need to know which source created a datum, which transform created a feature, which model produced an inference, which policy allowed the effect, and which evidence verified the outcome.

Existing standards cover parts of this. OpenTelemetry is useful for model, agent and tool spans. OpenLineage models jobs, runs and datasets. W3C PROV describes entities, activities and agents. They overlap, but they answer different questions. None of them grants permission or proves causal value.

The reference system is well tested. The live system has not earned that claim.

By the time I froze the architecture, the reference package had 683 automated tests, 220 defect records and 13 seeded unsafe mutants detected. Those numbers helped find real bugs. They do not prove the university portal, OpenClaw runtime, browser session or deletion path works in production.

A large test suite can make a weak claim look stronger than one complete live trace. I would rather keep the release state at HOLD than blur the difference.

One read-only workflow gets to earn trust first

The first production-shaped Autonomy Cell is a UON material-change monitor. It checks every current course source, detects changed obligations, keeps the evidence, and interrupts me only when something material changes or the state cannot be established. The review is pre-registered, and each gate has an exit condition.

1. Freeze the real source contract. List every current course, portal, page, pagination rule, freshness rule and authentication state. A source is either checked or explicitly unresolved. Exit condition: every expected source is accounted for.
2. Build the deterministic baseline. Use APIs or Playwright first. Capture authoritative snapshots and compare them without model interpretation wherever possible. Exit condition: missing data can never produce “no change”.
3. Operate in shadow. Schedule real checks, survive session expiry and crashes, and record model, browser, verifier, latency and cost receipts. Exit condition: 28 complete scheduled runs.
4. Calibrate the AI layers. Collect adversarial verifier negatives and labelled Jev operation-target decisions against the deterministic source of truth. Exit condition: 300 negative cases and 500 Jev labels.
5. Make the promotion decision. Review each action family using the pre-registered threshold and the exact version tuple that produced the evidence, and promote only the action families whose error bounds fit the risk budget. Exit condition: each action family gets one result, PROMOTE, HOLD or DEMOTE.

If it passes, the platform expands by outcome, not by data source

The likely sequence is GitHub and CI outcome tracking, calendar reconciliation, then PocketSmith read-only monitoring. OwnTracks and screen activity come later, after the system can connect context to a verified result.

1. GitHub and CI outcome graph. Connect tasks to commits, review, deployment health and product outcomes.
2. Calendar reconciliation. Prove reversible writes, conflict ownership and authoritative readback.
3. PocketSmith monitor. Keep financial context behind its own sensitive-data boundary.
4. Later, personal context. Add location and activity only for a named workflow with a measured benefit.

That order avoids building a detailed surveillance system that cannot tell whether anything useful happened.

My current rule is plain: build one bounded outcome, prove it in the world and keep the parts that earned trust. The model can say “done”, but the system must prove what happened. That is the difference between a demo and something you can leave running.

OUTCOME CONTRACT
- Freeze the requested external state.
- Enumerate required sources and forbidden effects.

AUTHORITY
- Issue a short-lived capability for exact operations, origins, objects, quota and expiry.
- Never infer authority from model confidence.

EXECUTION
- Prefer API or deterministic software.
- Use Playwright for stable GUI paths.
- Use Jev for bounded local ambiguity.
- Use a frontier model only for novelty or recovery.

VERIFICATION
- Read back authoritative state.
- Preserve UNKNOWN after ambiguous writes.
- Return a machine-readable receipt.

PROMOTION
- Promote only action families whose local error bounds fit their risk budget.