How do I know my agent's tools are not silently failing
The short answer. You stop asking the report and start comparing records. Before the work, a commitment goes into an append-only journal the agent cannot edit later: the job, the deadline, the acceptance test. During the work, every call mints a verifiable receipt at execution time, written by the executing system, not by the agent's summary. After the deadline, the comparison runs itself: commitment, claimed output, receipt timestamp. Three records, three moments, two writers. A silent failure is any place they disagree, and disagreement is a fact about records, not an impression about tone.
Scope, stated once and applied everywhere on this page: a verifiable receipt proves the saved result was not changed since it was saved. It does not prove the result was correct, and it does not prove an external event happened beyond what the record itself shows. The journal proves a commitment was recorded at a time and has not been edited since. Together they prove whether the promised work produced a matching record by the deadline, which is exactly the question a silent failure hides.
Why tool failures go silent
The failure is structural, and it has four common shapes. The swallowed error: the tool returns an error, the harness or the model absorbs it, and the step is summarized from the plan instead of the outcome. The empty result read as a clean result: a query that matched nothing comes back as a list of zero items, and zero items gets written up as checked, nothing found, which is a different claim from the one the empty result supports. The timeout summarized from the request: nothing came back at all, and the summary describes what was asked as though it were what was learned. The call that never ran: the step was skipped, deferred, or routed to a tool that does not exist, and the paragraph about it was written anyway.
All four shapes share one property: the final answer contains no marker separating them from success. The summary is generated text, and generated text about a failed call and generated text about a completed one are the same genre, same confidence, same formatting. Asking the agent whether its tools failed is asking the suspect to grade its own homework, in prose. That is why monitoring the transcript never catches this class of failure, and why sterner instructions do not fix it. The information about whether the call ran does not live in the transcript. It lives in the records the executing system wrote, or did not write, at the time.
The sibling failures have their own pages: the invented result is covered in How do I catch an AI agent making up results?, and the invented completion in AI agent lying about done. This page covers the quietest one: the work that failed without the agent, or anyone, noticing.
The fix: commit before work, then compare three records
- Write the commitment first. Before the job starts, log it to an append-only journal: what will be done, in words a stranger could judge, the deadline it will be done by, and the acceptance test that decides done. Append-only matters: a commitment that can be edited after the fact is just another summary, and a commitment written after the work is a rationalization, not a commitment.
- Run the work through recorded calls. Each call the job depends on should mint a verifiable receipt as it executes: recorded inputs, saved output, timestamp, provenance class, digests committed at that moment. The receipt is the unit the comparison runs on, so the calls that matter must be the calls that leave one.
- Let the deadline pass. The acceptance test needs a moment when it is judged, and that moment is the deadline in the commitment. Judging early invites in-progress excuses; the journal's whole point is that the test was fixed in advance.
- Compare the three records. The journal entry, the output the agent claims, and the receipts with their timestamps. No interview, no re-reading the chat for vibes. The records either line up or they do not.
- Read the outcome off the comparison. All three consistent: the work happened, on time, as claimed. A receipt that landed after the deadline: the work happened late, whatever the summary says. A claim with no receipt: the step did not run through a recorded call, and its result is unverified. A journal entry with no claim and no receipt: the work was abandoned silently, and now it is not silent. No journal entry at all: the job was never committed, which is the first finding, before any question of execution.
Notice what the pattern never does: it never asks the agent to explain itself. Explanations are more generated text. The comparison is between artifacts written before, during, and after the work, by systems with no stake in the summary. That is what makes a catch mechanical instead of rhetorical, and it is the same discipline as the completion check in Did my AI agent really complete the task?, run per job instead of per run.
What each failure mode looks like against the records
- The swallowed error shows up as a journal entry and a confident claim, with a receipt whose saved output is the error itself, or no receipt for that step at all. The error was real; only its reporting was absorbed. The record unburies it.
- The empty result shows up in the receipt's saved output: the call ran and returned nothing, committed under a digest. Nothing found is a legitimate result when the record shows the search actually happened. Without the record, nothing found and never looked are indistinguishable, and that indistinguishability is the failure.
- The timeout shows up as a missing receipt inside the deadline window. Calls that complete leave records with timestamps; a step whose only trace is the sentence claiming it has told you when it happened: never.
- The skipped call shows up as a gap in the sequence: commitments and neighboring receipts on both sides, and a hole where this step's record should be. Whole jobs close under one record, covered in the whole-job receipt page, and a hole in the sequence is visible precisely because the rest of the chain is intact.
In every case the catch is the same shape: a specific, checkable absence or mismatch, located in time, attributable to a step. Compare that with the alternative, a growing suspicion that the agent's reports cannot be trusted, with no way to say where. The records turn distrust into findings, and findings into fixes: wire the receipt into that call, add the journal step to that job, shorten that deadline.
The format underneath: AER-1
The receipts used in every section above follow one open format: AER-1, IETF Internet-Draft revision 23 (draft-zambo-aer1-23). AER-1 is an IETF Internet-Draft, not a finalized standard. It defines the per-step verifiable receipt and the workflow record that links receipts for a run, so a receipt produced in one place can be checked by a receiver somewhere else, with no account on the system that produced it and no cooperation from the operator who ran the agent. The journal discipline on this page is the operating pattern around the format: commitments before, receipts during, comparison after.
To put receipts under your own agent's work: start with the live demo for a single checked call, then a Zambo Day Pass for a day of calls with a verifiable receipt under each one. If you already have a run you suspect, the Unverifiable Run Clinic works the case with you.
Questions, answered
How do I know my agent's tools are not silently failing?
You cannot know from the agent's report, because the report is generated text and a swallowed failure reads exactly like a success. You know from records the run did not write about itself: a commitment logged to an append-only journal before the work, stating what will be done, by when, and what counts as done, and a verifiable receipt minted by each call as it executes. After the deadline, compare the three: the commitment, the claimed output, and the receipt with its timestamp. Work that happened has all three, consistent. A silent failure is the gap between them, and the gap is mechanical to spot.
What is a silent tool failure?
A silent tool failure is a tool call that errors, times out, returns empty, or never runs, while the agent's summary reports the step as done. The failure is silent because nothing in the final answer marks it: the model writes the summary from intent and partial context, and a failed call and a successful one produce the same confident sentence. The only trace of the difference lives outside the summary, in whether a record of the call exists and what it saved.
Why does an agent report success when a tool call failed?
Because reporting is generation, and generation completes the pattern. The agent planned the step, attempted it or meant to, and then wrote the summary the plan implies. Error text returned by a tool can be skimmed past, an empty result can be read as a clean result, and a timeout can be summarized from the request instead of the response. None of this needs intent to deceive. It needs only a summary written by the same process that did the work, which is why the check has to run on records written by a different process: the executing system, at execution time.
What is the commit-before-work pattern?
Before the work starts, the agent writes a commitment to an append-only journal it cannot edit afterward: the job it will do, the deadline it will do it by, and the acceptance test that decides done. The work then runs through calls that each mint a verifiable receipt. When the deadline passes, anyone can compare the journal entry, the output the agent claims, and the receipt timestamps. The commitment is what converts a silent failure from a suspicion into a finding: the work was promised in writing, in advance, and the record either shows it landed or shows it did not.
Does this catch an agent that did the work badly?
It catches work that did not happen, happened late, or produced a different result than the one reported. It does not grade quality. A verifiable receipt proves the saved result was not changed since it was saved; it does not prove the result was correct, and the journal proves a commitment was made and whether a matching receipt landed by the deadline. Judging whether the delivered work is good is still a review task. What the pattern removes is the prior question, whether there is any work there to judge at all.
What if my AI agent fails silently overnight?
Then the morning report is the one artifact you cannot use: it is generated by the same process that failed. What you can use is the pair of records this page sets up before the night starts: the commitment written to the append-only journal (the job, the deadline, the acceptance test) and the receipts minted as calls execute. Compare them at the deadline. A job that ran has a commitment and matching receipts inside it. A job that failed silently has the commitment and a gap, and the gap, not a vibe about the summary, is the finding. The worked example on this page ran exactly that comparison in public: journal job silent-failure-demo-20261015, commitment logged 23:00:16 UTC, receipt d0a8ff1d-fddb-45de-92b4-05bc1aa561ff landed 18 seconds later, inside the 23:30 deadline.
How do I know what my agent did overnight?
Read the records it could not rewrite. The journal shows what was committed before the run, with timestamps. The receipts show which calls actually executed, what they returned, and when, each one checkable at its verify link without an account. Anything the agent claims beyond those two records is unverified by definition, including the parts of the morning summary that sound the most confident. If the work matters, the overnight rule is simple: no receipt, it did not provably happen.