My AI made up results. Here is how to catch it.

How do I catch an AI agent making up results

The short answer: the claim-to-record test. Take the specific result the agent stated. Not the whole answer, not the summary: the number, the quote, the finding. Then ask one question of it: where is the record of the call that produced this? A result that was actually produced has one, minted at execution time, and the record's saved output is the result itself, committed under a digest. The test has exactly three outcomes, and each one tells you what you are holding:

• The saved output matches the claim, and the digest recomputes. The call ran, and the result in front of you is the result it produced, unchanged. This is the only outcome that earns trust, and it costs the agent nothing when the work was real.
• A record exists, but its saved output differs from the claim. Something ran, and then the claim drifted from it: rounded into a different number, upgraded from a partial result, or rewritten outright. The mismatch is the catch, and it is mechanical, not arguable.
• There is no record. No call produced the result. Whatever its source, it was not a tool returning data; it was the model producing text that resembles a result. Treat it as unverified, because unverifiable is what it is.

Scope, stated once and applied everywhere on this page: a verifiable receipt proves the saved result was not changed since it was saved. It does not prove the result was correct, and it does not prove an external event happened beyond what the record itself shows. For this problem that scope is the whole game: the test separates results a call actually returned from text the model composed, and it proves the returned ones reached you intact.

Why agents make up results

Not usually out of anything like intent. The failure is structural. An agent's reply is generated text, and generated text has no built-in marker separating a tool returned this from this is what a tool would probably return. When a call fails, times out, was never available, or was quietly skipped, the model still has to finish the paragraph, and finishing the paragraph fluently is the thing it is best at. The invented result inherits the tone of the real ones around it: same format, same confidence, same citations style. That is why proofreading the answer never catches it. A fabricated price and a fetched price read identically, because reading is not the layer where they differ. They differ at the record layer: one of them has a receipt and the other has nothing.

This is also why the fix cannot be a sterner prompt. Telling the agent not to invent results adds an instruction to the same text channel that produced the invention. The fix is to route the work through calls that leave records, so that every result arrives with its evidence attached and the claim-to-record test can be run by anyone, including the agent's own operator, in seconds. The lying-about-done page covers the sibling problem, the invented completion, and Did my AI agent actually run it? covers the per-call version of the question.

The check, step by step

1. Isolate the claim. Pick the exact result you are about to rely on: the figure, the quote, the yes or no. Vague answers cannot be checked; specific claims can.
2. Ask for the record, not a screenshot. The receipt for the call, as a link to the public record. A screenshot is an image of text and can be produced as easily as the claim was.
3. Compare the saved output to the claim. Open the record and read its saved result. Word for word, digit for digit. This is the step where fabrications and quiet alterations surface.
4. Recompute, or let the verifier do it. The output digest is computed over the saved bytes at execution time. The public verifier at zambo.dev/verify recomputes the commitment, payload, provenance, and metadata checks in the open; a passing set means the record in front of you is the record that was minted.
5. Read the provenance and the time. The receipt states the provenance class, who ran or reported the call, and the execution timestamp. A result claimed from this morning with a record from last week is a mismatch of a quieter kind.
6. Act on the outcome. Match: proceed. Mismatch: the claim is wrong, whatever the run produced. No record: the result is unverified, and it stays that way until it is rerun through a recorded path.

The whole test takes under a minute per claim, needs no account, and works the same whether the agent ran in Claude, ChatGPT, Gemini, Cursor, or Grok, because the record format does not change with the client.

What a record that settles it looks like

Not every artifact called a record settles anything. The ones that do share a shape:

• Minted at execution time by the system that ran the call, not written up afterward by the agent or its operator.
• The recorded inputs, committed under an input hash, so the record shows what the call was actually asked.
• The saved result, committed under an output digest, the exact bytes the comparison runs against.
• The provenance class, stating who ran or reported the action, bound into the same record.
• The tool, its version and scope, and the execution timestamp, so the record says what executed and when.
• A public location, so the person checking needs no account, no permission, and no cooperation from whoever ran the agent.

Hold any result's evidence up against that list. The missing items are usually the finding: a record written afterward, a result with no digest, a claim whose only location is the chat itself. Each gap is a place where text can be, and eventually will be, composed instead of computed.

When the citation itself is fabricated

A fabricated citation is the made-up result wearing a reference, and it comes in three kinds. The source does not exist: the link is invented, the paper was never published, the page returns a 404, the quote appears nowhere in the document it is attributed to. The source exists but does not say that: a real link carrying a claim its text never makes, counted on nobody opening it. The source exists and says something close: and the claim quietly upgrades it, a may into a does, a draft into a standard, one figure into a rounder one. All three survive proofreading, because a citation looks like diligence. None of them survives the source check.

The check, complete, in order:

1. Resolve the source. Open the cited link, or fetch the page. A source that does not resolve, never existed, or resolves to a different document than the one described ends the check on the spot: the citation was fabricated, and every claim resting on it is unsupported until it is re-grounded.
2. Match the claim to the text. Read what the source actually says and compare it with the claim, word for word on the parts that matter: the number, the date, the yes or no. The claim is backed only where the text is. A real source under an invented claim is the same failure one layer down, and this is the step that catches it.
3. Keep the lookup receipt. Run the fetch through a recorded call, so the text you checked is committed under a digest at the moment you checked it. Pages change, drafts are revised, links rot. The lookup receipt freezes what the source said when the claim was judged, so the citation can be re-examined later against the same text, not against whatever the page says by then.

The habit that prevents all three kinds: citations are produced by lookups, not by memory. When the agent fetches the source through a recorded call and writes the claim from the saved output, the citation arrives with its evidence attached, and the check above takes seconds because the record already exists.

The format underneath: AER-1

The receipts used in every section above follow one open format: AER-1, IETF Internet-Draft revision 23 (draft-zambo-aer1-23). AER-1 is an IETF Internet-Draft, not a finalized standard. It defines the per-step verifiable receipt and the workflow record that links receipts for a run, so a receipt produced in one place can be checked by a receiver somewhere else, with no account on the system that produced it and no cooperation from the operator who ran the agent.

To put receipts under your own agent's work: start with the live demo for a single checked call, then a Zambo Day Pass for a day of calls with a verifiable receipt under each one. The general method this page applies is laid out in Verify AI work: check the record, not the summary.

Questions, answered