I do not need a leaderboard. I need to know whether last Tuesday's agent change quietly broke the checkout flow for anyone on a preview URL.

That is the eval problem for people who ship UI. If you cannot catch product damage, you do not have an agent improvement. You have a dice roll with better commit messages.

Not BLEU. Not a PDF about judge models. A repeatable check that answers:

Did this agent / prompt / tool / model swap make the product worse for a real user journey?

If your answer is "we looked at two chats and they felt fine", that is not an eval. That is optimism.

Steal the software loop you already trust. Wrap it around agent changes.

Artefacts worth treating as code:

• instructions.md / system prompts

• tool schemas and allowlists

• skills

• model IDs and routing rules

• MCP connector versions

If it can change behaviour, it belongs in version control and it deserves a gate.

Playwright (or your stack's equivalent) against a preview:

• sign in

• land on the page the agent claims it fixed

• assert the button exists, the error toast does not, the network call returned the shape you expect

Agent says "fixed empty state". Eval opens the empty state path and screenshots the truth.

When the agent learns a new tool, freeze golden inputs and expected structured outputs:

UI teams already understand contract tests. Apply them to the agent boundary.

Agents love rewriting JSX. They are less loyal to focus order and contrast. A small axe + screenshot diff on the routes in the change set catches the "it works but it is hostile" class of failure.

Wire the forge CLI into the loop. An agent change that breaks the pipeline is already evaluated. Believe the pipeline before you believe the chat transcript.

For each agent-assisted PR I want three boxes ticked:

Check: Journey | Pass looks like: Critical path script green on preview

Check: Contract | Pass looks like: Tool schemas + golden cases unchanged or intentionally versioned

Check: Regressions | Pass looks like: No new a11y/visual failures on touched routes

Fail any one, and the prompt does not ship. Yes, even if the demo in Slack looked gorgeous.

Evals will not catch every product sin. They will not replace design critique. They will not stop an agent from implementing the wrong ticket brilliantly.

They will stop you from confusing "model feels smarter" with "users can still complete billing".

Also: keep evals cheap enough that people run them. A two-hour ML-flavoured harness nobody executes is theatre. A four-minute Playwright slice on preview is culture.

MCP and typed tools reduce malformed actions. DevTools and browser tools improve observation. Eve gives you a directory you can version. Katie Knows taught us graph RAG quality shows up in eval pass rates when the representation is right.

This post is the unfashionable glue: measure the product, not the prose.

Ship the eval that protects the UI, or admit the agent is allowed to break the product in production silence.