The fairness protocol, in five steps
Most agent comparisons are rigged before they start. Each agent does a slightly different task, reports in its own words, and gets judged on confidence. The fix is to hold everything constant except the agent:
- One task statement. The same sentence, given to both agents, specific enough that a stranger could score it.
- Checks written before either run. What the saved result must show. Written once, applied to both, unchanged after the results arrive.
- The same recorded tools. Both agents run their calls through one recording layer, so each run leaves a verifiable receipt trail in the same format. Through the Zambo MCP at https://zambo.dev/mcp, every call returns its result plus a verifiable receipt at a public link.
- Score the saved records side by side. Checks passed, calls used, time taken, cost. Score the records, not the summaries, and ideally without knowing which trail came from which agent until the scoring is done.
- Keep both trails with the scores. Anyone should be able to reopen the two records and re-derive the verdict.
Zambo is the verifiable work layer for AI: 153 tools behind one endpoint, and every call receipted under AER-1, published as IETF Internet-Draft draft-zambo-aer1-26. That receipt trail is what makes steps 3 to 5 possible at all.
Can I compare two AI agents on the same task fairly?
Yes, under the protocol above. The comparison is fair exactly when the task, the checks, and the recording are shared, because then the only thing left to differ is the work itself. It turns unfair the moment any of the three is per-agent: a reworded task, checks adjusted after seeing one agent's output, or one agent self-reporting while the other is recorded. If you take one habit from this page, take the checks-first habit. It is the step everyone skips and the step that decides whether the result means anything. To run a comparison this way, add Zambo to your AI: 20 free calls per day, then they upgrade when the job needs more.
Worked example: one real recorded run, scored
This run was executed live through https://zambo.dev/mcp and you can open the record yourself. The task, stated before the run: return the current live price of BTC in USD, with the 24 hour change and the market cap, from a live source. The checks, written before the run: the result names BTC and a USD price; it carries a 24 hour change; it carries a market cap; it carries a live timestamp close to the run time; the run leaves a public record a second person can re-score.
The recorded run
A live_price call at 2026-10-23T15:03:57Z. Saved result: BTC $81,652.00 USD, down 0.10% over 24 hours, market cap $1,641.43B, live timestamp matching the run time. The call left verifiable receipt a233dfc7-ac28-4d4c-87e9-2b7d0499a775. Open it, or check it at zambo.dev/verify/.
The score, and how a second agent joins the table
All five checks pass against the saved record: score 5 of 5. Now run a second agent on the identical task through the same endpoint. Its calls leave the same shape of record, so the same five checks score it without adjustment, and the two trails sit side by side: checks passed, calls used, cost. That is the whole trick. There is no special comparison machinery, just one shared record format and the discipline to write the checks first.
What metrics measure AI agent performance?
Read them off the record, in four groups. Task success: checks passed over checks total, per task. The primary metric; everything else is supporting detail. Accuracy: saved values against a known correct value, or output against the expected shape fixed before the run. Efficiency: tool calls used, elapsed time, and cost, all present in the receipt trail. Reliability: across repeated runs of the same task, the share that pass the same unchanged checks. Two habits keep the metrics honest: fix the set before the first run, and keep the record beside every score so the number can always be traced back to evidence.
How do I benchmark AI coding agents against each other?
Same protocol, task-shaped checks. The benchmark task names the files, the behavior, and the tests. Each agent runs it through the recording layer. Scoring reads the saved record: does the saved diff touch only the named files, do the saved test results show the named tests passing, did the run produce the required artifacts, how many calls and how much time did it take. The failure mode this removes is self-report bias, the benchmark quietly becoming a comparison of two agents' descriptions of their work. The full setup, including how the merge gate reads the same record, is on verifying an AI-generated code change before merging.
How do I evaluate whether my AI agent did a good job?
That is the single-agent version of this page's question, and it has a complete method page of its own: how to evaluate whether your AI agent did a good job. The short form: task first, checks second, recorded run third, score last, and never grade the agent's summary of itself. The comparison protocol on this page is what that method becomes when a second agent enters.
The verifier-scale example: the AER-1 conformance corpus
A fair comparison needs a fixed target everyone is scored against. AER-1 ships one in public: a conformance corpus of 126 vectors and 129 checks, kit v1.5.7, published at zambo.dev/aer1/corpus.json and archived on Zenodo under DOI 10.5281/zenodo.23677115. Any verifier can run the same vectors and the results are comparable because the inputs and the expected outcomes were fixed in advance. Steal that shape for your own benchmarks: publish the task and the checks with the run, keep the inputs fixed, and make the scoring repeatable by a stranger.
Honest limits: what a receipt does not do
Read this before you rely on any of it. A verifiable receipt proves the saved result was not changed since it was saved. It never proves the result was good, correct, or wise. Zambo is not an eval platform: no leaderboards, no automated quality scores, no general eval suites. It is the evidence layer underneath your own evaluation. A fair comparison also only covers the tasks you actually ran, scored by the checks you actually wrote.
When a full day of comparison runs is the job, the paid rungs are plain: a $0.99 Day Pass for 24 hours; the Zambo Pass at $29 for 30 days, one payment, or $290 for 365 days; and $19 for 30 days, one payment for $ZAMBO holders.
Questions, answered
How do I evaluate whether my AI agent did a good job?
Evaluate the saved record, not the agent's summary. State the task in one sentence, write the checks the result must pass before the run, run the agent with its tool calls recorded, then score the saved results against the checks one by one and keep the record with the score. A verifiable receipt for each call proves the saved result was not changed since it was saved, so anyone can re-derive your score from the same evidence. Whether the work was good stays your judgment, made against that record. The full method has its own page on this hub.
What metrics measure AI agent performance?
Four groups, all readable from a saved record. Task success: checks passed over checks total, the primary metric. Accuracy: saved values compared with a known correct value or the expected shape you specified before the run. Efficiency: tool calls used, elapsed time, and cost, all in the receipt trail. Reliability: the share of repeated runs of the same task that pass the same unchanged checks. Fix the metric set before the first run, so a bad result cannot quietly change what counts.
How do I benchmark AI coding agents against each other?
Give every agent the same stated task, the same checks written before any run, and the same recorded tools, then score each saved result against those checks. For coding work the checks are task-shaped: the saved diff touches only the named files, the saved test results show the named tests passing, the required artifacts exist. Run each agent's calls through one recording layer such as the Zambo MCP at https://zambo.dev/mcp, so every run leaves a verifiable receipt trail in the same format, and compare the trails side by side: checks passed, calls used, time, and cost.
What is an AI agent eval?
An AI agent eval is a repeatable test of agent work: a stated task, checks written before the run, a recorded run, and a score derived from the saved results against those checks. Eval is short for evaluation. A good eval can be re-run by a stranger from the same record and produce the same score, because the evidence is kept instead of summarized. An eval is not the agent grading itself and it is not a feeling about the output. It is a method: task first, checks second, recorded run third, score last.
How do I score AI agent output quality automatically?
Automate the evidence and the mechanical checks, not the judgment. Record every tool call so the saved result is fixed, then run scripted checks against that record: did the call happen, did the output match the expected shape, did a required field hold the expected value, did a stated test pass. Those return pass or fail with no human in the loop. What automation cannot honestly do is judge whether the result was good, wise, or fit for the purpose. That stays with the evaluator who wrote the checks. Zambo supplies the recorded evidence the mechanical checks run against. It does not score quality for you.
Can I compare two AI agents on the same task fairly?
Yes, if both agents get the same task, the same checks written before either run, and the same recorded tools, and you score both saved results against those checks. Fairness fails when each agent reports its own work in its own words, because then you are comparing two summaries, not two results. Run both through the same recording layer so each run leaves a verifiable receipt trail, then place the trails side by side and apply the same checks to both. The shared record removes self-report bias. It does not remove judgment: the same evaluator should score both records.