AI browser agent

Context

I wanted the mechanism, not a demo that wraps a browser in a prompt and hopes. The interesting part of an agent that uses a browser is the boundary. Who is allowed to act, what they can see, and how a person notices before something careless becomes something expensive.

This is a learning build. No users, no uptime number, no benchmark. The point is a design I can draw and then defend.

Constraints

The agent does not inherit a daily browser profile. Cookies, extensions, and logged-in sessions stay out unless they are placed there for the task.

Tools are a list, not a paragraph in the prompt. If a capability is not on the list, the model does not get to invent it.

A run has a budget. Steps, time, and a stop state. An open loop that keeps trying is how the plot gets lost.

A person should see the run while it happens. A transcript after the fact is a weaker instrument.

Architecture

The shape is small on purpose. Five pieces, and the arrows between them are the design.

1. 01ModelProposes one tool call from the goal and the latest observation.
2. 02OrchestratorHolds the budget, the state, and the right to stop.
3. 03MCP allowlistTurns a proposal into a typed call, or refuses it.
4. 04PlaywrightRuns the call in a browser context that dies with the run.
5. 05Run logStreams each event, in order, to a person watching.

The model proposes. The orchestrator decides whether that proposal is still inside the budget. MCP is the edge where a proposal becomes a typed call, or gets refused. Playwright runs in a context that dies with the run. The run log is the surface a person actually looks at: events, in order, not a hidden console.

Decisions

Playwright, not raw Chrome DevTools Protocol, for the first cut. That choice hides some of the wire protocol. I wanted a browser I could launch, isolate, and throw away before spending the week on framing. The protocol can be a later incision.

One browser context per run, with a temporary user-data directory. When the run ends, the directory goes with it. Persistence would have to be added. It is not a default I have to remember to turn off.

The tool list is short: navigate, snapshot, click, type, and done. Snapshot means an accessibility tree, not a screenshot. Structure is cheaper to reason about than pixels, and it fails in ways you can see.

MCP exposes that list. The model talks to a server with an allowlist. Importing a function inside the loop, and hoping the prompt mentions it, is a weaker boundary. A protocol makes the list something you can read.

The orchestrator is a small state machine. Propose, validate, act, observe. It stops on done, on budget, or on a tool error it will not retry blindly.

Data flow

A goal arrives as text. The orchestrator sends the goal plus the current observation to the model. The model returns a tool call, or it says it is finished.

The call is buffered until it is complete. Partial arguments are not executed. The UI can show that a proposal is in progress. It does not pretend the action already happened.

MCP checks the name and the arguments against the allowlist. Playwright performs the action inside the ephemeral context and returns a new observation: URL, title, and a trimmed accessibility snapshot. The trim matters. A footer repeated across three pages will fill a context window with nothing.

That observation is appended to the run log and streamed out. The next proposal sees the observation, not a paraphrase of it. Then the loop continues, or it stops.

Tradeoffs

Accessibility snapshots are a poor fit for canvas, maps, and anything that only means something as pixels. I accept the miss. A screenshot can be a later tool, explicit and expensive, not the default eye.

A strict allowlist makes the agent less magical. That is the point. Magic is what you call a side effect you cannot list.

Ephemeral contexts mean the agent cannot casually ride an existing login. A task that needs an account needs a session created for the task. Inconvenient, and much easier to explain.

Playwright's idea of idle and a single-page app's idea of idle are not the same. Network idle is a hint, not a proof that the page is ready.

Problems

If the observation is sloppy, the model narrates actions it did not take. A sterner prompt does not fix that. An observation format does: URL, a short snapshot, and the exact tool result, so a lie is obvious on the next line of the log.

Streaming tool arguments as they generate is tempting and messy. I wait until the call is well formed. The log then has three beats: proposing, the call, the result.

Timeouts are a product decision disguised as a number. Too short, and slow pages look broken. Too long, and a stuck run looks thoughtful. The budget is visible in the log so a stall looks like a stall.

What I learned

Isolation is the design. It is not a flag you flip after the demo works on your own machine.

The protocol boundary is what makes the tool list real. If I cannot point at the server and read the allowlist, I do not have one.

Step budgets teach more than clever instructions. A bounded loop with a dull log is more honest than an unbounded agent that sometimes looks brilliant.

Next iteration

Approval when navigation leaves the origin the run started on. The log already has the URL. The missing piece is a pause a person can answer.

Replay from the event log, so a run can be read without driving the browser again. If the log is the source of truth, replay is just taking that seriously.