How to audit AI agent tool calls: capture at call time, audit by recompute.

Why the transcript is not an audit

Most agent auditing today means reading the transcript carefully, or handing it to a second model to grade. Both approaches have the same flaw: the transcript is the agent's own narration. It was produced by the system being audited, from that system's account of its own run. Reading it harder does not change who wrote it.

• It can omit. A call that failed, a retry, a step that was skipped: if the narration leaves it out, there is nothing in the transcript to notice the absence with.
• It can reorder and smooth. Narration describes the plan as much as the execution, and a summary written after the fact reads the same whether the run was clean or chaotic.
• Nothing in it can be recomputed. A transcript carries no commitments. There is no hash to check, no bytes to re-derive, so the strongest verdict available is that the story sounds plausible.
• The grader grades a story. A second model reviewing the transcript produces a second opinion about the narration, not evidence about the run.

An audit needs material the narrator does not control. That material is a per-call record made during the run, by the layer that executed the call, committed under hashes the agent cannot go back and adjust.

Step 1: capture the record at call time

The record has to be made when the call happens, by the executing layer, not reconstructed later from logs the agent assembles. Per tool call, the receipt captures:

• The tool name that actually ran, not the one the plan mentioned.
• The arguments, committed twice over: a canonical serialization of the exact argument bytes, and a SHA-256 hash of the receipt's canonical bytes, so any later edit to the recorded inputs breaks the commitment.
• The observed result: the exact bytes that came back from the call, saved, with their hash. Not a paraphrase of the result and not the agent's summary of it.
• A timestamp for when the call executed, which is what makes stalls and gaps measurable later.
• The call's place in the run: each step receipt carries its position, so a multi-step job can be bound into a single workflow record with one root commitment over the ordered set of steps.

Storage is part of the method. The receipts live outside the agent's control, in a store the agent cannot edit, reorder, or delete. A record the audited system can rewrite is a diary, and auditing a diary is just reading it.

Step 2: audit by recompute

With the record captured, the audit is mechanical, and it is the same for one call or a thousand:

1. Recompute every commitment. For each receipt, compute the SHA-256 over its canonical bytes and compare it to the output hash the receipt carries. A match means the saved record is the record that was made at execution time. A mismatch is a finding, full stop.
2. Check the inputs against the job. The recorded tool and arguments are right there in the receipt. Confirm they are the tool and the arguments that step should have used. This is where wrong-recipient sends, wrong-file reads, and quietly substituted parameters surface.
3. Recompute the run as a set. For a multi-step job, recompute the workflow root from the step receipt IDs and compare it with the published root. Then count: steps in the job, steps in the record. A step the agent claims and the record lacks did not run, or ran somewhere that left no evidence, which for audit purposes is the same thing.
4. Read the world back. Take the effects the run claims (the written file, the updated row, the sent request) and read them back from the live system of record. Hashes prove the record is intact; the readback proves the record matches reality. The audit needs both.

The worked version of this procedure, run against a real public receipt with the hashes shown, is the receipt verifier walkthrough. The same checks run offline with the open conformance kit in the AER-1 kit repository.

Step 3: count what the record makes countable

Anecdotes do not improve a system; rates do. Once every call leaves a receipt, the failure modes that used to be vibes become numbers you can track per model version, per tool, per workflow:

• Bypass rate. Runs where a mandatory closing call never fired, or a required step has no receipt, divided by total runs. The record shows the absence directly.
• Leak and contamination rate. Calls whose recorded arguments contain material that belongs to the agent's internal reasoning rather than to the tool: visible because the arguments themselves are on the record, hashed and inspectable.
• Stall rate. Runs with a timestamp gap far beyond the job's normal step interval, and nothing in it. The walk-away test is the simple version of this check.
• Claim mismatch rate. Steps the summary claims that the record or the readback does not back. This is the number that tells you whether a new model version actually got more honest, or just more fluent.

None of these rates exist without the captured record. With it, an argument about whether an agent is reliable turns into a table anyone can recompute.

Run your first tool-call audit

• See one receipt: the live demo runs a real call and hands you its verifiable receipt, no signup. That receipt is the unit every step above operates on.
• Audit a whole run: connect an agent over MCP at zambo.dev/mcp and every call in the run leaves a receipt as it executes. A Day Pass ($0.99) keeps a full day of runs receipted and seals a closed run so its record stays put.
• Keep the receipts in an open format: the receipts on this page follow AER-1, an IETF Internet-Draft (draft-zambo-aer1-23), so the record stays checkable by anyone, with public test vectors and independent implementations, long after the run.