We publish the receipts.
Anyone can demo an agent on the case it was built for. The useful question is what it does on the case it was not — and almost nobody publishes that. Here is what every Veriphos agent goes through, and what you get to read afterwards.
The gauntlet
Seven stages. An agent that cannot clear all of them does not get listed.
Write the contract first
Before any prompt exists, the agent gets a written spec: what it is for, what it must never do, what it hands back to a human, and what counts as a correct answer. Everything downstream tests against this document.
Build the scenario suite
Real cases, drawn from real work — the ordinary ones the agent exists for, plus the awkward middle ground where judgement is actually required. Each scenario has an expected outcome written before the agent is run against it.
Attack it
An adversarial pass: prompt injection hidden in the content it reads, contradictory instructions, hostile and malformed input, and deliberate attempts to make it act outside its contract. An agent that can be talked out of its own rules has not passed.
Put a ceiling on cost and latency
An agent that is correct but takes forty seconds and a pound a run is not shippable. Every agent carries a p95 latency ceiling and a cost-per-run ceiling, and blowing either fails the gate as surely as a wrong answer.
Have a human read the failures
Automated scoring catches whether the output matched. It does not catch output that is technically correct and practically useless. Someone reads the failures and the near-misses, every release.
Keep every bug forever
Every defect ever found becomes a permanent regression scenario. The suite only grows. This is the part that stops an agent quietly getting worse as it changes.
Publish the report — pass or fail
The report goes in the bundle. If an agent misses its gate, it does not go in the catalogue, and we say so rather than shipping it with softer numbers.
What a release gate looks like
A gate is four numbers and a rule. The numbers are set when the agent is specified, not negotiated afterwards to fit whatever it happened to score.
- Scenario count — how much surface the suite covers. More is not automatically better; representative is better.
- Minimum pass rate — typically 98%, higher for agents that touch money or send on your behalf.
- p95 latency ceiling — the slow tail, not the average. Averages hide the experience people actually complain about.
- Cost-per-run ceiling — so an agent cannot buy accuracy with your budget.
Miss any one of the four and the release is blocked. Not shipped with an asterisk — blocked.
This is only meaningful if it costs us something, so: agents will sit in in testing on the catalogue for as long as it takes, and you will see them there.
What we will not claim
Veriphos is pre-release. No agent has completed its suite yet, so there are no measured results anywhere on this site — every number you see is a target we have set ourselves. The eval reports on the agent pages carry measured: null for exactly that reason, and they will keep carrying it until the suite has genuinely run.
When results do exist they will be published whole, including what failed. A report that only shows wins is marketing, not evidence.