How do you benchmark AI coding agents against each other when every vendor benchmark is run by a vendor?

How do I benchmark AI coding agents against each other?

Give both agents the identical task, the identical starting files, and the identical time budget. Then keep a receipt for every tool call each one makes, save every output exactly as produced, and compare the saved outputs plus the receipt stream side by side. The record is what makes the comparison fair and re-checkable. Without it, you are comparing two stories about work, not the work itself.

Scope, stated once: an agent receipt is a record of one ai agent tool call that anyone can check: it keeps the exact saved bytes and their sha-256 digest, and it proves the result was not changed since it was saved, not that it was right.

Vendor benchmarks are run by vendors. The task set, the scoring, and the runs that get published are all chosen by the seller. That does not make every vendor number false. It makes it uncheckable by you. A benchmark you run yourself, on your own task, with a full receipt stream for both agents, is smaller and far more useful: you can reopen every step and see exactly where one agent took a wrong turn and the other did not.

Can I compare two AI agents on the same task fairly?

Yes, if three things stay identical and provable. First, the input: the same prompt, the same repo state, the same files, the same constraints. Second, the conditions: the same tools available, the same time and call limits. Third, the record: a receipt for every call, so the steps each agent actually took are saved, not reconstructed from memory or from a summary the agent wrote about itself.

Fairness breaks in quiet ways. One agent gets a cleaner starting branch. One gets extra retries that never get mentioned. One edits a file that the other was never shown. A receipt stream catches those breaks, because the calls are listed in order with their saved results. If the inputs diverged, the stream shows it.

A fair head to head, step by step

1. Pick one real task. Use a task from your own work: fix this bug, add this endpoint, refactor this module. Real tasks expose real differences. Toy puzzles mostly expose puzzle practice.
2. Freeze the starting point. Same commit, same prompt text, same instructions, written down before either agent runs.
3. Run both through the same endpoint. With Zambo at https://zambo.dev/mcp, each tool call issues an AER-1 receipt, specified by the IETF Internet-Draft draft-zambo-aer1-26. The receipt stream becomes the shared score sheet.
4. Score the saved outputs, not the claims. Tests passed, diff size, files touched, calls used, time taken, and whether the result still passes when you rerun it a week later from the saved record alone.
5. Publish the stream with the verdict. Anyone who disagrees with your ranking can verify the receipts at zambo.dev/verify and point to the exact call where your reading is wrong. That is a benchmark other people can build on.

Worked example: a real receipt from a live call

From the public stream on October 25, 2026: a capability_search call saved at 2026-10-25T18:35:28Z, receipt at zambo.dev/run/48493b66-f9bf-4bff-80cb-052af5716698, saved result digest sha256:a2b6680939153af26134fbdd5aca53dc2544f8e8a2dcd9f1122975d585f30d4d.

The verification endpoint returned verified for this receipt. In a head to head, every call from both agents would carry a receipt like this one. You would not be trusting a final summary. You would be comparing two complete, checkable streams, call by call, and the losing agent could not quietly rewrite its steps after the fact, because the saved results were not changed since they were saved.

Try it: open the verifier and paste that receipt link. Then run your own first comparison with 20 free calls per day at zambo.dev/add.

What to score, and what to ignore

HANDOVER, Zambo flagship product, matters here too: when a benchmark run hands its full record to the next person or the next agent, the comparison can continue instead of starting over from a claim.