How we measure.
Eval is the product. Every capability is judged on labeled truth. This page shows the method, the numbers, and the receipts.
Two benchmarks bracket the range.
Two public sets run through the same engine that serves every client match. Each is around 200 rows and deliberately nasty. One reads like a messy real CRM export. One is torture built to break us. Run them both right here, live against the serving pack with no key needed.
Both numbers print together. The match rate tells you how much work the engine did. The wrong-entity rate counts the rows the engine matched to the wrong company, and tells you whether you can trust the work.
Both sets plant fake companies with no real counterpart on record. The no-match line counts the engine refusing to invent a match for them. A 100 percent refusal means zero hallucinated matches. An engine that never says no would match every one of them to the wrong company. That line catches it.
Run both public benchmarks yourself, live against the serving pack. No key needed. A representative CRM file shows field performance and an adversarial set shows the honest floor. Both are small and deliberately nasty. Results cache per reference pack so every run returns the identical receipts.
Around 200 rows of real entities, each hit with the nastiest single noise class we found in the wild: the first two words of the name merged into one with the space deleted. No domains. Planted fabricated no-match rows keep the false-match rate measurable. A low match rate here is the set doing its job. It measures the floor under the worst noise we have seen and keeps the wrong-entity rate honest while recall work proceeds.
200 rows composed to read like a real client CRM export. Fields fill at the rates measured on a real 20-year export: domains on 83 percent of rows, city on 60, postal on 55. The name noise is modeled on that export's documented badness at the magnitudes it actually showed, from legal-form drift to single-character typos. Planted no-match traps carry plausible countries and cities so the trap is fair. Two rows swap their real domain for a placeholder whose real owner is a different company. An engine that follows the placeholder scores a wrong entity honestly. The rows are seeded synthesis on public reference data. No client data was read to build it and the same pack and seed rebuild the identical artifact.
Numbers report as-is on any pack. No benchmark is ever special-cased.
Every run returns a receipt.
Every run returns a receipt: a sha256 computed over the per-row dispositions. The receipt binds the run to the set version and the reference pack it scored against. Identical inputs produce the identical receipt on any deploy. We verified a benchmark receipt byte-identical across three production deploys in one day on 2026-07-14. Anyone can rerun the public sets and diff the receipts so two parties compare numbers without trusting either one.
3 production deploys · 2026-07-14 · byte-identical
Confidence the labels earned.
A matching tier ships as an automatic match only when its human-labeled sample reads at or above 95 percent precision. Tiers below the bar are held behind corroboration guards. This is the label batch that governs the deployed engine.
The honest hold
One tier read 78 percent on its labeled sample. It did not ship as auto. It serves behind a corroboration guard that requires independent evidence before an automatic match. The label program is allowed to say no. That is what calibration means.
The per-tier bars come from these human labels. The per-row confidences carry a further calibration fit on 667,381 held-out proof pairs (cal-v1), whose worst measured over-claim is zero.
The measurement record carries the statistical test behind the bar, and why the bar moved.
Why we bias toward the miss.
A row we leave unmatched stays yours to resolve and is never billed. A row we merge into the wrong company corrupts every decision made downstream of it and destroys trust in every attribute we return, so under-merge is the safe error and every guard in the engine carries that bias.
When the evidence is thin it refuses. When two entities sit genuinely close it routes the row to review. It never guesses to lift a score. Every benchmark on this page prints its wrong-entity rate beside the match rate, because a confident wrong answer is the one that costs you.
Leak-free, or the number means nothing.
A benchmark number is only worth the discipline behind it. Every eval we run clears these checks before its number counts for anything.
Hold out the signal being graded.
When we grade the name path the strong key that would label the pair is held out of the matcher's inputs. The engine earns the match on the evidence it would see in production.
Score against the company. Never the row.
When a prediction lands on a sibling record of the same company, it counts as a correct link. Duplicate rows of one company were never separate answers to get wrong.
No circular truth.
Truth minted from the artifact under test is invalid. When we have caught ourselves doing it we have thrown the number away and re-baselined. The cases are written up in the measurement record.
Grade retrieval against the whole corpus.
Every match is graded against the full Master Entity List with millions of distractors in the pool. A pairwise score never stands in for corpus recall.
Version-lock the truth to the pack.
Every truth set is rebuilt against the same spine version as the pack under test so the entity id spaces line up and a stale label can never flatter a fresh pack.
We record negative results and park them in dated docs. A parked negative is a result.
The measurement record carries the cases, including a holdout we discarded and re-baselined lower.
Run the benchmarks yourself.
The public endpoint answers anonymously, no key needed. Point an agent at it or paste this into a terminal.
curl -s https://turntodata.ai/api/eval/public?set=crm_v1