The evaluation method, in four steps
Most agent evaluation goes wrong at the first step: the run happens first and the criteria get invented afterward, usually by reading the agent's own summary and nodding along. Reverse the order and the whole problem gets smaller.
- State the task before the run. One sentence, specific enough that a stranger could tell whether it was done. "Get the current BTC price in USD" is a task. "Research crypto" is not.
- Write the checks before the run. List what the saved result must show for the work to count as good: the right fields, the right shape, values inside a stated range, a source and a time attached. Checks written after you see the output are not checks. They are rationalizations.
- Run the agent with its tool calls recorded. Route the work through a recording layer so each call leaves a record at the moment it executes. Through the Zambo MCP at https://zambo.dev/mcp, every call returns its result plus a verifiable receipt at a public link.
- Score the saved results against the checks, and keep the record. Go check by check, pass or fail, against the saved output. Keep the record with the score so anyone can re-derive the score later from the same evidence.
That is the whole method. Everything else on this page is detail: which metrics to read off the record, how to compare two agents fairly, what can be automated, and where the method stops.
How do I evaluate whether my AI agent did a good job?
Judge the saved record, not the summary. Write the task and the pass checks before the run, record every call while it runs, then score the saved results against those checks one by one and keep the score with the record. A receipt proves the saved result was not changed, not that the work was good, so the judgment stays yours and stays checkable by anyone who reopens the record. To run a scored trial, add Zambo to your AI and use the 20 free calls per day to record the first job.
How do I score AI agent output quality automatically?
Automate the checks that can be checked by rule: required fields present, values in range, files and links that open, and outputs that match the format the task asked for. Run those checks against the saved results in the AER-1 receipts, count passes and failures per job, and leave human judgment for the parts rules cannot judge, such as tone and fit. Zambo records the calls behind the score, across 153 tools with 35 on the default surface, so an automatic score can always be traced back to the work it scored. Open the receipt verifier to see the saved record a score is built from.
Worked example: scoring a real recorded job
This example was run live for this page on October 23, 2026, and you can open the record yourself and repeat the scoring.
Step 1: the task, stated before the run
Task: return the current live price of BTC in USD, with the 24 hour change and the market cap, from a live source rather than from training data.
Step 2: the checks, written before the run
Check 1: the result names BTC and a price in USD. Check 2: the result carries a 24 hour change figure. Check 3: the result carries a market cap figure. Check 4: the result carries a live timestamp close to the run time, not a stale date. Check 5: the run leaves a public record a second person can open and re-score.
Step 3: the recorded run
The job ran as a live_price call through https://zambo.dev/mcp at 2026-10-23T15:03:57Z. The saved result was: BTC $81,652.00 USD, down 0.10% over 24 hours, market cap $1,641.43B, with a live timestamp of Fri, 23 Oct 2026 15:03:57 GMT. The call left verifiable receipt a233dfc7-ac28-4d4c-87e9-2b7d0499a775, with output digest sha256:07688fec8f514088482b5265b910db1c4329d03674af5591d35fda9fa03bb778. Open the receipt, or check it at zambo.dev/verify/.
Step 4: the score, derived from the saved record
Check 1: pass, the saved result names BTC at $81,652.00 USD. Check 2: pass, a 24 hour change of 0.10% is in the record. Check 3: pass, a market cap of $1,641.43B is in the record. Check 4: pass, the timestamp in the record matches the run time. Check 5: pass, the receipt link above is public and re-scorable. Score: 5 of 5 checks passed. Notice what the score means and what it does not. It means this saved result satisfied the five checks written for this task. The receipt proves the saved result was not changed since it was saved. It does not prove the price was the best obtainable price, and no receipt can prove that. That judgment belongs to whoever wrote the checks.
Worked example: scoring a call that was rejected
An evaluation that only scores successes is marketing. This second example, also from October 23, 2026, scores a failure: a journal_log call sent without its required arguments.
The record
The call was rejected before execution: error code invalid_tool_arguments, missing required arguments "job_id" and "tool_name". It left verifiable receipt cf4eda14-167f-440c-bec5-439f5e2e60c2, typed pre_execution_rejection and saved at 2026-10-23T19:05:05Z. Open it: the page is titled "Rejected call: journal_log", in plain words.
The score
Score it with the same discipline. Task completion: fail. No result was produced, and the record says so instead of smoothing it over. Record completeness: pass. The rejection, its reason, and its time are all in the saved record. An honest evaluation keeps both facts at once: the agent did not do the job, the record proves it did not, and there is no result anyone could bill for. A summary written by the agent itself could quietly drop this call. A receipt trail cannot: the rejection is part of the record, and the record is what you score.
What metrics measure AI agent performance?
Read every metric off the saved record, never off the agent's summary. Four groups cover most real evaluations:
| Metric group | How to score it from the record |
|---|---|
| Task success | Checks passed divided by checks total, per task. The primary metric. Everything else is supporting detail. |
| Accuracy | Saved values compared with a known correct value, or output compared with the expected shape you specified before the run. |
| Efficiency | Tool calls used, elapsed time, and cost, all readable from the receipt trail for the run. |
| Reliability | The same task run repeatedly: the share of runs that pass the same unchanged checks. |
Two habits keep metrics honest. First, fix the metric set before the first run, so a bad run cannot quietly change what counts. Second, keep the record beside the metric, so a score can always be traced back to the evidence it was derived from.
Receipt based metrics such as verified completion rate, calls per job, and rework rate all read off that same saved record. For the client facing side of these numbers, see showing a client every step your AI took.
Comparing two agents fairly, and benchmarking coding agents
A fair comparison needs three things held constant: the same task, the same checks written before either run, and the same recorded tools. Give both agents the identical task through the same recording layer, so each run leaves a verifiable receipt trail in the same format. Then place the two trails side by side and apply the same checks to both records: checks passed, calls used, time taken, cost.
For coding agents the pattern is the same, with task-shaped checks: does the saved diff touch only the files the task named, do the saved test results show the stated tests passing, does the output include the artifacts the task required. Score the saved code and saved test output in the record. The failure mode this removes is self-report bias: without a shared record you end up comparing two agents' descriptions of their work, which measures writing confidence, not work quality.
What fairness cannot remove is judgment. The same evaluator should apply the same checks to both records, and disagreements about a check should be settled by rewriting the check, not by re-arguing the run.
What can be automated, and what cannot
The honest split matters, because this is the question where overclaiming is easiest. You can automate the evidence collection: record every call, fix the saved result at execution time, and keep the trail. You can automate the mechanical checks: did the call happen, did the output match the expected shape, did a required field hold the expected value, did a stated test pass. Those return pass or fail with no human in the loop, and they should run against the recorded evidence so the result can be checked later.
You cannot automate the judgment that the result was good. Whether a summary is insightful, whether code is well designed, whether a recommendation fits the situation: those calls stay with the evaluator who wrote the checks. Any tool that claims to score general agent quality automatically is selling the judgment part without doing it. Zambo does not do that part. It supplies the recorded evidence, and the mechanical checks run against that evidence.
A public benchmark you can re-run yourself: the AER-1 conformance corpus
The method above has a worked public example at verifier scale. AER-1, published as IETF Internet-Draft draft-zambo-aer1-26, ships a public conformance corpus that anyone can download and run a verifier against: 126 vectors and 129 checks, kit v1.5.7, read live from zambo.dev/aer1/corpus.json on October 23, 2026. The corpus works exactly like a good eval should: the expected outcomes are published in advance, the inputs are fixed, and anyone can re-run the checks and re-derive the result. Two verifiers that both pass the full corpus agree on the record format in a way no summary can fake.
Use it as a template for your own evaluations: publish the task and the checks with the run, keep the inputs fixed, and make the scoring repeatable by a stranger.
Honest limits: what a receipt does not do
Read this before you rely on any of it. A verifiable receipt proves the saved result was not changed since it was saved. It never proves the result was good, correct, complete, or wise. Quality judgment stays with the evaluator's checks, always. Zambo is not an eval platform: it does not provide benchmark leaderboards, it does not provide automated quality scores, and it does not provide eval suites for general agents. What Zambo provides is the evidence layer underneath your own evaluation: a recorded, checkable trail for work that runs through it, so the scoring you do rests on the real saved output instead of on the agent's account of itself.
Three more limits worth stating plainly. A recorded trail only covers work that ran through the recording layer, so route the run through it before the work starts; a receipt cannot be created honestly after the fact. Checks can only test what they name, so a 5 of 5 score means five stated things held, nothing more. And a fair comparison compares the runs you recorded, on the tasks you stated; it says nothing about tasks you did not test.
Run your own evaluations on recorded work
Start with one recorded trial in Zambo Chat, no signup, and score it with the four steps above. When a full day of evaluation runs is the job, the paid rungs are plain: a $0.99 Day Pass for 24 hours covers a day of recorded trials; the Zambo Pass is $29 for 30 days, one payment, or $290 for 365 days; and it is $19 for 30 days for $ZAMBO holders. Every call through https://zambo.dev/mcp leaves the verifiable receipt your checks will score against. More on the receipt itself: AI agent receipts, what they are and how to check one.
Questions, answered
How do I evaluate whether my AI agent did a good job?
Evaluate an AI agent by scoring the saved record of its work against checks you wrote before the run, not by reading the agent's own summary. State the task in one sentence, list the checks the result must pass, run the agent with its tool calls recorded, then score the saved results against the checks one by one. Keep the record so anyone can re-derive your score from the same evidence. A verifiable receipt for each call proves the saved result was not changed since it was saved, so the score rests on the real output. Whether the work was good stays your judgment, made against that record.
What is an AI agent eval?
An AI agent eval is a repeatable test of agent work: a stated task, checks written before the run, a recorded run, and a score derived from the saved results against those checks. The word eval is short for evaluation. A good eval can be re-run by someone else from the same record and produce the same score, because the evidence is kept instead of summarized. An eval is not a feeling about the output and it is not the agent grading itself. It is a method: task first, checks second, recorded run third, score last.
What metrics measure AI agent performance?
The metrics that measure AI agent performance fall into four groups you can score from a saved record. Task success: did the saved result pass each check you wrote before the run, counted as checks passed over checks total. Accuracy: did the values in the result match a known correct value or an expected shape. Efficiency: how many tool calls, how much time, and how much cost the run used to reach the result. Reliability: across repeated runs of the same task, how often the result passes the same checks. Pick the metrics before the run, read them off the record after the run, and keep the record with the score.
How do I benchmark AI coding agents against each other?
Benchmark AI coding agents by giving every agent the same stated task, the same checks written before any run, and the same recorded tools, then scoring each saved result against those checks. The shared task and shared checks are what make it a benchmark instead of two separate demos. Run each agent's tool calls through a recording layer such as the Zambo MCP at https://zambo.dev/mcp so each run leaves a verifiable receipt trail, then compare the trails side by side: checks passed, calls used, time taken, and cost. Score the saved code and test output in the record, not the agent's description of its own work.
How do I score AI agent output quality automatically?
You can automate the evidence collection and the mechanical checks, not the judgment. Automate this part: record every tool call so the saved result is fixed, then run scripted checks against that record, such as did the call happen, did the output match the expected shape, did a required field hold the expected value, and did a test suite pass. Those checks return pass or fail without a human in the loop. What automation cannot honestly do is judge whether the result was good, wise, or fit for the purpose. That judgment stays with the evaluator who wrote the checks. Zambo does not score quality automatically. It supplies the recorded evidence the mechanical checks run against.
Can I compare two AI agents on the same task fairly?
Yes, if both agents get the same task, the same checks written before either run, and the same recorded tools, and you score both saved results against those checks. Fairness fails when each agent reports its own work in its own words, because then you are comparing two summaries, not two results. Run both agents through the same recording layer so each run leaves a verifiable receipt trail, then place the two trails side by side and score them with the same checks. The shared record removes self-report bias. It does not remove judgment: the same person should apply the same checks to both records.
Related
- Verifying the identity of an AI agent
- When two AI agents disagree about what happened, what record settles it?
- The receipt stream: every agent job, receipted in public
- AI agent observability: how do I track what my AI agent is doing in real time?
- ZVEB: the benchmark where every score has a receipt, on zambo.dev