How do I evaluate whether my AI agent did a good job?

Every evaluation argument in agent work collapses into one missing thing: a record of what the agent actually did. Teams score vibes from a summary the agent wrote about itself, then wonder why the scores do not predict failures. Settle the facts first, then judge quality on top of them.

Two questions hiding inside one

Did the agent do the job? That is factual: the calls it made, the order, the results, the finish. And was the work good? That is judgment. Evals, rubrics, and graders address the second. The first is answered by the job's record, and in Zambo every finished job carries a verifiable AER-1 receipt: open it and the steps are there, each result hashed, the whole record checkable by anyone. When the first question is settled by evidence, the second one gets much cheaper to answer.

What is an AI agent eval, and where does it stop?

An eval runs an agent against a set of tasks and scores the run: useful for choosing a model, catching regressions, and comparing versions in general. Its limit is scope. An eval score says how an agent does on average; it says nothing about the specific job a client paid for on Tuesday. That question is per-job, and per-job questions need per-job proof: a receipt for that job, not a benchmark for that model.

How do I score AI agent output quality automatically?

Start by making completion machine-checkable. Bind the work into a job, journal every tool call as it happens, and wrap the job when it ends. The wrap runs a proof loop over the job's own steps, so the receipt attests to a record that was checked, not a claim that was typed. Here is a real one, from a live agent session: https://zambo.dev/run/3d23c2f2-abcf-488e-ba0f-80e2eb9b4def. With existence and completion scored from the record, human and model graders spend their time on the part that actually needs judgment: whether the output was any good.

Can I compare two AI agents on the same task fairly?

Yes, if both work against the same record standard. Zambo jobs carry a per-agent caller identity, so two agents doing the same task produce two receipts that can be compared line by line: steps, tools, duration, and the sealed result. That is also how a second AI picks up and checks the first one's work, through a handover, instead of starting over on trust.

Related

Questions, answered

How do I evaluate whether my AI agent did a good job?

Split it into two questions. First, did the agent do the job at all: which tools ran, what came back, in what order. That is settled per job by a verifiable receipt. Second, was the output good: that is a judgment, made by a human, a rubric, or an eval set. Most teams skip the first question and argue about the second with no record to argue from.

What is an AI agent eval?

An eval is a test set run against an agent to score its behavior across many tasks: accuracy, completion rate, cost. Evals measure an agent in general. They do not settle what happened in one specific job yesterday, which is what a client, a manager, or a dispute actually asks about. Per-job proof and evals answer different questions, and serious setups keep both.

How do I score AI agent output quality automatically?

Automate the checkable part first: bind the job, journal every call, and verify each step's saved result against its receipt, so completion is scored from the record instead of the agent's summary. Quality scoring, rubric or model-graded, then runs on outputs you have already confirmed are real.

Can I compare two AI agents on the same task fairly?

Give both the same job record to work against and compare receipts, not summaries: steps taken, tools used, time, and the verified result each one sealed. Two agents' self-reports are not comparable; two receipts are, because both are checked the same way.

Run your first job free

Zambo is the verifiable work layer for AI: tools behind one MCP endpoint, continuity that carries work across AIs, and a verifiable AER-1 receipt for every job. Connect the endpoint at https://zambo.dev/mcp and the first 20 calls plus one wrap-up are free every day. Picking up a job in another AI is always free. A Day Pass is $0.99 for unlimited calls for 24 hours.