The benchmark that publishes its own losses.
When it links records automatically, it's right 99.8% of the time. Anything uncertain goes to human review, never to guesswork, because in company data, a wrong match costs more than a missed one.
No established public benchmark measures company-record entity resolution. ER-7 is built to be that benchmark. The rules were frozen before any number existed, the metric is the false-merge rate rather than a blended score, and the datasets ship CC0 with everything needed to regenerate them.
Read the false merges first.
Read the false-merge rate first.
FM@R is the share of linked pairs that joined two genuinely different companies, read at a pinned recall. A missed link leaves the row yours to resolve. A false merge corrupts every decision downstream of it, which is why the wrong answer prints ahead of the work done.
No F1 appears anywhere.
One blended number is marketing. F1 averages the two error costs as if they were equal, and in company data they are not. The report publishes the asymmetry instead of hiding it in a mean.
Abstention is a feature.
Every system may abstain and an abstention scores as a non-link. The served engine additionally routes hard pairs to human review and publishes that rate. Routing a coin-flip to a person instead of forcing a guess is the design claim under test.
Losses publish at full prominence.
The rules were frozen before any number existed and the two defects found since are printed below with their corrected numbers. A benchmark that only ships when it wins is marketing.
The table is sectioned for a reason.
The systems below do not all solve the same problem, so the results are printed as two tables and no row should be compared across that line.
the pairwise systems
exact_key, Splink, dedupe and AWS Entity Resolution see two name strings plus the pair's country and decide: same entity or not. That is classical record linkage and it is all these tools are asked to do.
the reference-assisted product
The turntodata rows measure the product. The served matcher resolves each side against the TurnToData Master Entity List, around 224 million entities built from open sources, and links when both sides resolve to the same entity. The Master Entity List ingests GLEIF, the corpus this benchmark is generated from, and its other-entity names are index features. On clean pairs the turntodata row measures reference-assisted resolution on home turf. That reference access is the product being sold, and the comparison is deliberately product versus tool.
the reference advantage, measured before our numbers exist
zero 27.33% · below one third 3.82% · at or above one third 68.85%
zero 33.95% · below one third 4.29% · at or above one third 61.76%
Zero shared tokens. Transliterations, renames, alternate-language legal names. No pairwise name matcher can link these. A reference-assisted resolver can, by lookup.
The name-matchable mass. The textured sliver is barely string-matchable, overlap below one third.
On v1.2 name evidence is all a pairwise system has. The datasets carry no address columns and country is constant within every pair by construction. A pairwise name matcher's recall ceiling on clean pairs is therefore 72.67 percent, the non-zero-overlap mass. A pairwise system can still exceed the ceiling by linking indiscriminately, and the table shows what that costs. Any turntodata recall above the ceiling at useful precision is reference lookup at work.
fairness, both directions
The baselines received a fixed, logged tuning budget on the train split. The served config is not tuned on this benchmark at all and predates the split. That asymmetry favors the baselines.
The reverse asymmetries are larger and favor us. The matcher was developed against evaluations drawn by the same generator doctrine from the same source corpus, and the served system consults a reference that ingests the answer-generating source while every other system sees only the pair. Both are stated here rather than discovered by a reader.
False merges at pinned recall.
FM@R50, R70 and R90 on heldout, per system per dataset. Full per-run tables with config hashes, PR curves and slices live in the harness output beside the report.
| system | dataset | abstain | max recall | FM@R50 | FM@R70 | FM@R90 |
|---|---|---|---|---|---|---|
| exact_key | pairs v1.2 | 0 | 0.094 | no R50 | no R50 | no R50 |
| exact_key | damaged v1.2 | 0 | 0.033 | no R50 | no R50 | no R50 |
| splink | pairs v1.2 | 0 | 1.0 | 0.586 | 0.586 | 0.586 |
| splink | damaged v1.2 | 0 | 1.0 | 0.586 | 0.586 | 0.586 |
| dedupe | pairs v1.2 | 0 | 1.0 | 0.455 | 0.532 | 0.532 |
| dedupe | damaged v1.2 | 0 | 1.0 | 0.510 | 0.555 | 0.555 |
A system that cannot reach a pinned recall has no FM@R point to publish, and the ceiling itself is the result. Splink reaches R50 only by linking every row shown, so its false-merge rate at every pinned recall is the negative base rate 0.586. The trap negatives are same-country name collisions by design and no pairwise name signal separates them.
aws entity resolution · quarantined at its v1-measured subsamples
| regime | dataset | abstain | max recall |
|---|---|---|---|
| aws_er_ml | pairs 20k subsample, v1 | 0.00005 | 0.382 |
| aws_er_ml | damaged 20k subsample, v1 | 0.00005 | 0.359 |
| aws_er_rule_based | pairs 20k subsample, v1 | 0 | 0.031 |
| aws_er_rule_based | damaged 20k subsample, v1 | 0 | 0.007 |
The AWS arm runs on seeded 20,000-pair stratified subsamples of v1, quarantined at those measured rows rather than mixed into the v1.2 table. The managed ML matcher cannot reach R50 on either set. Its recall ceiling is 38.2 percent on pairs and 35.9 percent on damaged, so the ceiling itself is the result.
The documented-rule regime measured recall ceilings of 3.1 percent and 0.7 percent. A rule ladder that requires exact field agreement cannot see name variants at all. That is the exact-key floor wearing a managed-service badge.
Total AWS spend across all attempts is $180, stated once. The first rule-based attempt failed on our own record-id format. That was our error and it is printed here, and the funded redo processed every record.
the same table, read as a picture · heldout, both datasets
dedupe 0.455 · 0.532 · 0.532 · splink 0.586 flat
dedupe 0.510 · 0.555 · 0.555 · splink 0.586 flat
The bars are dedupe, the one pairwise system with a real operating curve. The dashed line is splink at every pinned recall, because its first operating point at or above R50 links every row shown. exact_key and both AWS regimes never reach R50, so they have no point to draw and their ceilings are the result. Values print exactly as the table above states them.
| system | dataset | abstain | max recall | FM@R50 | FM@R70 | FM@R90 |
|---|---|---|---|---|---|---|
| turntodata (served posture) | pairs v1.2 | 0.358 | 0.744 | 0.0021 | 0.0021 | no R50 |
| turntodata (served posture) | damaged v1.2 | 0.802 | 0.217 | no R50 | no R50 | no R50 |
Read plainly: the exact-key floor cannot reach one row in ten on clean pairs, which is the case for entity resolution in one line. The supervised baseline has a real operating curve and still merges 45 percent of what it links at R50 on clean pairs and 51 percent on damaged. On sets built from nothing but public GLEIF names, pairwise matching at useful recall merges roughly half of what it links. That is the structural argument for reference-assisted resolution and for routing hard pairs to review instead of forcing a guess. Our own row gets this same treatment when it lands, wherever the numbers fall.
Two defects found and printed.
Both were found after the v1 numbers landed, by measurement. Both fixes shipped in the v1.2 regeneration and the corrected numbers replaced the old rows. The frozen methodology carries the full dated addendum, because frozen text gains addenda and is never rewritten.
The address artifact
Independent review measured a designed-in artifact in v1. The generator copied each LEI's single legal address to both sides of every positive, so the rule "link when city and postal are equal" scored recall 1.0 at a false-merge rate of 0.167 on pairs and 0.172 on damaged with no name comparison at all. It beat both tuned baselines, and both had partially ridden the same artifact.
The fix. v1.2 drops the address columns from both published sets. No address rule can score on data where the columns do not exist, so the artifact class is closed structurally rather than by tuning. The measurement instrument publishes with the benchmark and refuses to run on v1.2.
The train contamination
GLEIF carries duplicate registrations, two issued LEIs for one organization with identical names. At the published content grain that put 1,631 pairs train rows and 6,146 damaged train rows string-identical to heldout content, every pairs row inside the false-merge trap class. The supervised baseline had trained on copies of its own test traps, so part of its measured trap resistance was memorization.
The fix. v1.2 drops any train row whose published content matches a heldout row. Heldout is untouched, so no published measurement moved. The baselines re-tuned on clean train and the corrected numbers replaced the v1 rows.
what the fixes moved
| baseline | dataset | v1 FM@R50 | v1.2 FM@R50 |
|---|---|---|---|
| dedupe | pairs | 0.119 | 0.455 |
| dedupe | damaged | 0.146 | 0.510 |
| splink | pairs | 0.548 | 0.586 |
| splink | damaged | 0.261 | 0.586 |
Dedupe's headline moved from 0.119 to 0.455 on pairs and from 0.146 to 0.510 on damaged when the two defects were fixed. Splink moved from 0.548 to the 0.586 base rate. The earlier numbers looked better for every baseline and the improvement was ours to take away. Both finds publish here with the same prominence as any win because the pre-registered rules require it.
The set that did not exist.
No established public benchmark measures company-record entity resolution. The candidate families were verified against the frozen methodology's rule and the exclusion reasoning publishes with the benchmark.
10,398 negative_exact rows share a standardized name key across conflicting LEIs. That is the canonical false-merge trap. 24,907 negative_near rows share a distinctive token.
The pairs rows under five perturbation classes: drop_token, word_swap, initial_truncate, strip_punct_legal, char_typo. Robustness to real client damage is the dimension existing benchmarks skip.
ttd-gleif-pairs is built to be that benchmark. It ships CC0 with its generator, seed and split hash so anyone can regenerate it, verify it or extend it.
Reproduce every number.
One command per system per dataset from a clean clone. Metric tables are byte-identical run to run. If a third party cannot reproduce a number, that number does not publish.
uv run python scripts/build_er7_datasets.py \
--gleif <golden_copy.parquet> --out data/er7_benchmark_v1_2 --seed 20260712
ER7_DATE=2026-07-17 bash harness/run_<system>.sh \
--data <dataset>/heldout.parquet --train <dataset>/train.parquet \
--dataset-version <dataset>/1.2.0 --out harness/out/<system>/<name>Eval is the product.
The measurement doctrine behind this page, the live public sets and the per-tier calibration program are published in full.