Measurement
How we measure.
Eval is the product. Every capability is judged on labeled truth. This page shows the method, the numbers, and the receipts.
The cardinal sin
False merge is the cardinal sin.
Merging two genuinely different companies destroys truth and the client's trust in every value we return. Under-merge is the safe error. A row we leave unmatched costs a lookup. A row we merge into the wrong company corrupts every decision made downstream of it.
Every guard in the engine carries that bias. When the evidence is thin it refuses. When two entities sit genuinely close it routes the row to review. It never guesses to lift a score. The wrong-entity rate is printed first on every benchmark below, ahead of the match rate, because a confident wrong answer is the one that costs you.
The honest range
Two benchmarks bracket the range.
Two public sets run through the same engine that serves every client match. Run them both right here, live against the serving pack, no key needed.
Read the wrong-entity rate first. It counts the rows the engine matched to the wrong company, and the run beside this recomputes it live against the serving pack. The match rate tells you how much work the engine did. The wrong-entity rate tells you whether you can trust the work.
Run both public benchmarks yourself, live against the serving pack. No key needed. A representative CRM file shows field performance, and an adversarial set shows the honest floor. Results are cached per reference pack, so every run returns the identical receipts.
Around 200 rows of real entities, each hit with the nastiest single noise class we found in the wild: the first two words of the name merged into one, the space deleted. No domains. Planted fabricated no-match rows keep the false-match rate measurable. A low match rate here is the set doing its job. It measures the floor under the worst noise we have seen and keeps the wrong-entity rate honest while recall work proceeds.
200 rows composed to read like a real client CRM export. Fields fill at the rates measured on a real 20-year export: domains on 83 percent of rows, city on 60, postal on 55. The name noise is modeled on that export's documented badness at the magnitudes it actually showed, from legal-form drift to single-character typos. Planted no-match traps carry plausible countries and cities so the trap is fair. Two rows swap their real domain for a placeholder whose real owner is a different company, and an engine that follows the placeholder scores a wrong entity honestly. The rows are seeded synthesis on public reference data. No client data was read to build it, and the same pack and seed rebuild the identical artifact.
Numbers report as-is on any pack. No benchmark is ever special-cased.
Receipts
Every run returns a receipt.
Every run returns a receipt: a sha256 computed over the per-row dispositions. The receipt binds the run to the set version and the reference pack it scored against. Identical inputs produce the identical receipt on any deploy. We verified a benchmark receipt byte-identical across three production deploys in one day, 2026-07-14. Anyone can rerun the public sets and diff the receipts, so two parties compare numbers without trusting either one.
3 production deploys · 2026-07-14 · byte-identical
The label program
Confidence the labels earned.
A matching tier ships as an automatic match only when its human-labeled sample reads at or above 95 percent precision. Tiers below the bar are held behind corroboration guards. This is the label batch that governs the deployed engine.
The honest hold
One tier read 78 percent on its labeled sample. It did not ship as auto. It serves behind a corroboration guard that requires independent evidence before an automatic match. The label program is allowed to say no. That is what calibration means.
The per-tier bars come from these human labels. The per-row confidences carry a further calibration fit on 667,381 held-out proof pairs (cal-v1, active), whose worst measured over-claim is zero.
The discipline
Leak-free, or the number means nothing.
A benchmark number is only worth the discipline behind it. Every eval we run clears these checks before its number counts for anything.
Hold out the signal being graded.
When we grade the name path, the strong key that would label the pair is held out of the matcher's inputs. The engine earns the match on the evidence it would see in production.
Score against the company, never the row.
When a prediction lands on a sibling record of the same company, it counts as a correct link. Duplicate rows of one company were never separate answers to get wrong.
No circular truth.
Truth minted from the artifact under test is invalid. We caught a domain holdout reading 92.7 percent off entities whose identity came from the very pages being re-crawled, discarded it, and re-baselined at 89.
Grade retrieval against the whole corpus.
Every match is graded against the full reference set with millions of distractors in the pool. A pairwise score never stands in for corpus recall.
Version-lock the truth to the pack.
Every truth set is rebuilt against the same spine version as the pack under test, so the entity id spaces line up and a stale label can never flatter a fresh pack.
We record negative results and park them in dated docs. A parked negative is a result.
Run it yourself
Run the benchmarks yourself.
The public endpoint answers anonymously, no key needed. Point an agent at it or paste this into a terminal.
curl -s https://turntodata.ai/api/eval/public?set=crm_v1