Open challenge · AER-1 conformance corpus, kit 1.5.7

Can your verifier pass 146 checks?

Every agent framework now claims its work is verifiable. Almost none of them publish a test that could prove their verifier wrong. AER-1 does, and this page is the standing invitation: run the corpus against your own verifier and publish the score.

Here is the failure this challenge exists to catch. Your framework issues a receipt for a finished job. A client, a judge, or another agent checks it with a verifier, and the verifier says valid. But the verifier was written by the same team, tested against its own outputs, and never once run against an outside test set. If its digest math is off by one byte of framing, if its chain binding accepts a record it should reject, every check it has ever run is a coin flip wearing a green checkmark. Nobody finds out, because nothing ever disagreed with it.

A standard only means something when independent code can fail its tests. So the AER-1 conformance corpus is public, versioned, and digest-pinned, and the challenge is simple: if you ship a verifier, run it against the corpus and post what it scores.

The corpus

How a run works

1. Fetch the index: https://zambo.dev/aer1/corpus.json
2. Pull each file from the base URL the index names, and confirm each
   file's SHA-256 against the digest printed in the index.
3. Run YOUR verifier over every vector. Not the reference kit: yours.
4. Compare all 146 checks, including the must-fail vectors, whose
   correct result is rejection.
5. Publish the score anywhere public, and link this page.
6. Claim your row in the official registry (next section): a passing
   verifier takes its place in the public record.

Claim your row

A public score is the run. The registry is where it lands. Passing implementations and verifiers are listed in the official AER-1 registry, the standing public record of who has actually run the corpus, and claiming a row takes one of two paths:

• Merge request: open one against the rambozambodotdev/rambozambodotdev.gitlab.io project adding your row to registry.json. Implementations take a six-field row in the implementations list (language, status, score, command, url, note), and independent verifiers are recorded in the validators list. The registry's Register Your Implementation section spells out the format.
• No GitLab account: reply to the implementer thread on The Colony with your language, score, repo link, and run command, and your row will be filed for you.

Every listed row was earned the same way: a public run against the public corpus. Pass in public, and the next builder who wonders whether AER-1 is real finds your name in the record and reruns your numbers.

What passing means, and what it does not

Passing means your verifier does the standard's math exactly: it recomputes the digest over the exact saved output bytes, it follows the chain from receipt to receipt, and it rejects a record whose saved result was changed. That is the whole claim, and it is a strong one, because it is the property every receipt you will ever check stands on.

Passing does not mean Zambo endorses your product, and it does not mean the work inside any receipt was good work. A verifiable AER-1 receipt proves the saved result was not changed since it was saved. It never proves the result was right. Conformance proves your checker enforces the first fact the same way the standard does. Anyone who tells you a receipt proves more than that is selling you the second claim with the first one's evidence.

It has been done from the text alone

This challenge is not theoretical. Independent verifiers, built by people with no connection to Zambo, have passed the corpus working from the published draft text alone, with no access to the reference implementation's internals. Their names are not this page's to hand out: the draft's own record carries them, and the official AER-1 registry lists the implementations that pass the kit. That is what a standard spreading looks like: strangers, public tests, recomputed numbers.

Every new public pass does the same work again. The next builder who wonders whether AER-1 is real finds your score, reruns the corpus, and gets the same 146. Post yours.

Related

• The conformance test, explained: what each check covers
• The implementation report: independent implementations and their conformance results
• What is AER-1?
• The official AER-1 registry

Frequently asked questions