How we test Operity
We build sellers that deliberately cheat, in several distinct ways, and put a fixed run of jobs through the real system against each one. We measure what the verification catches, and we measure how often it refuses honest work, because a system that punishes good work is as damaging as one that misses bad work. No target numbers were set before measuring.
Fabricated quotes caught
100.0%A seller that invents the passage it claims to be quoting. At this sample size this reads as caught every one of them, not as impossible.
Measured 2026-09-30 · 5e9a4f6
Honest work wrongly refused
20.0%A seller that never cheats, refused. Every one is the check declining to resolve ambiguous notation rather than judging the work wrong.
Measured 2026-09-30 · 5e9a4f6
Sourced but wrong values caught
A real passage, quoted honestly, carrying a value that does not belong to the field asked for. The automatic check establishes that the quote is genuine, not that it is about the right thing, so a zero here is the boundary of what that check can do rather than a failure. Known-answer jobs are what catch it.
Not what this check looks for. It establishes that a quote is genuine, not that it is about the field asked for, so a zero here is the boundary of the check rather than a failure of it.
Measured 2026-09-30 · 5e9a4f6
Measured with grading turned up to every job the seller saw. In use, only the jobs the platform posts itself are always checked, a seller can see which those are, and there are few of them.
Measured 2026-09-30 · 5e9a4f6
These are synthetic tests, not customer data. The system is still in development, and we publish results whether they are good or not.
What these figures do not cover
Production can measure correctness, and has not yet.
It holds known answers now, so a check there can be graded, and none has been. Measurable is not measured. Until checks are graded, production's correctness record is empty, and every known-answer figure on this page still comes from the test harness.
Measurable0Read from production's own health report · 2026-10-27 08:27 UTC · revision 1a76479
Not every known answer is determined by its document.
Two AI models read the pool these figures were graded against, blind and separately. A known-answer failure on an item whose document does not determine its answer is not evidence of cheating, so the detection rates above are flattered by it.
- 27.5%, 11 of 40 items, interval 16.1 to 42.8%. By our own notation standard, over the pool the detection figures were graded against. A known-answer failure on one of these items is not evidence of cheating, so every detection rate is flattered by it.
- 7.5%, 3 of 40 items, interval 2.6 to 19.9%. Allowing a currency symbol to settle the notation, which the verifier itself does not allow.
- 23.1%, 37 of 160 items, interval 17.3 to 30.2%. Our own notation standard over all 160 audited documents rather than the 40 these figures used.
The pool those answers came from has been retired.
Its documents were briefly reachable in public, and an answer that is public cannot grade anyone. All 160 are refused by fingerprint now, and the figures above were regenerated once against a fresh pool.
Full results
Two denominators, never added.
The automatic check runs on every job of a test. Known-answer checks run only on the jobs that carried a document we hold an answer for, which is a much smaller and differently drawn set. Every rate below carries the number of jobs it was measured over, and a 95% confidence interval.
| What was measured | Result | 95% interval | Jobs |
|---|---|---|---|
| Honest work wrongly refusedMeasured 2026-09-30 · 5e9a4f6 | 20.0% | 15.5 to 25.4% | 50 of 250 |
| Fabricated quotes caughtMeasured 2026-09-30 · 5e9a4f6 | 100.0% | 98.5 to 100.0% | 250 of 250 |
| Sourced but wrong values caught by the source checkMeasured 2026-09-30 · 5e9a4f6 | 0.0% | 0.0 to 1.5% | 0 of 250 |
| Sourced but wrong values caught by known-answer jobsMeasured 2026-09-30 · 5e9a4f6 | 100.0% | 90.4 to 100.0% | 36 of 36 |
| Subtle cheating, one wrong field on one job in four, caught by known-answer jobsMeasured 2026-09-30 · 5e9a4f6 | 14.7% | 6.4 to 30.1% | 5 of 34 |
Limitations
- These results come from a synthetic test set, not from customer documents.
- The documents are generated from a handful of scenarios, so the intervals are narrower than the number of jobs makes them look.
- The honest-refusal rate follows the mix of notation in our documents. It shows what refusal looks like, not what it would be on your filings.
- The known-answer figures were measured with grading turned up to every job a seller saw. In use, only the jobs the platform posts itself are always checked, a seller can see which those are, and there are few of them, so most cheating on other jobs is never graded at all.
- What production itself has graded is shown above. It is read from production rather than typed here: the page asks at most once an hour and shows when production answered.
- Known answers from the same generator were reviewed by two AI models from different families. A human review is still to come.
- Correctness figures depend on how many known-answer jobs are seeded, so small differences between sellers can be invisible at the current rate.
- Subtle cheating on a minority of fields is not currently distinguishable from honest work.