Entity Resolution at Scale: From Thousands to Millions of Records

Published February 2026 · 13 min read

Entity resolution is easy at small scale. With a thousand records, you can compare every pair (499,500 comparisons) in seconds. With ten thousand records, it's 50 million comparisons — still manageable.

With one million records, you need 500 billion comparisons. At one microsecond per comparison, that's 5.8 days of continuous computation. On a single core.

Scaling entity resolution isn't about raw compute power. It's about intelligent algorithms that reduce the work while maintaining match quality. This post covers the techniques that make entity resolution practical at scale: blocking, parallelization, incremental processing, and architectural patterns.

The Quadratic Wall

The fundamental challenge is that naive entity resolution is O(n^2). Every record must be compared to every other record.

Records	Comparisons	Time at 1μs/comparison
1,000	500K	0.5s
10,000	50M	50s
100,000	5B	83 min
1,000,000	500B	5.8 days
10,000,000	50T	1.6 years

No amount of hardware can brute-force the 10M record case. You need to reduce the number of comparisons.

Blocking: The Key to Scale

Blocking groups records into buckets using one or more attributes. Only records within the same block are compared. This reduces the comparison space from O(n^2) to O(n * b), where b is the average block size.

Blocking Strategies

Exact attribute blocking: Group records that share the same value for a field.

Block Key	Example Blocks	Avg Block Size	Reduction
Email domain	@acme.com, @bigco.com	50-500	~99%
Zip code	60601, 10001	100-1000	~99%
Last name	smith, jones, garcia	500-5000	~95%

Prefix blocking: Group records that share the first N characters of a field.

Block Key	Example	Reduction
First 3 chars of last name	"smi", "jon"	~97%
First 5 chars of email	"rsmit", "bsmit"	~99%

Phonetic blocking: Group records that sound alike.

Block Key	Example	Reduction
Soundex(last name)	S530 (Smith, Smyth)	~97%
Metaphone(first name)	RBR (Robert, Roberta)	~96%

Compound blocking: Combine multiple attributes.

Block Key	Example	Reduction
City + first initial	"chicago_r"	~99.5%
Birth year + gender	"1987_M"	~99%

Multi-Pass Blocking

No single blocking strategy catches all matches. A compound key like (zip + last name prefix) misses people who moved or whose name is misspelled.

The solution: multiple blocking passes with the union of all candidates.

Pass 1: Block on email → 50K candidates
Pass 2: Block on (last name prefix + zip) → 200K candidates
Pass 3: Block on phone number → 30K candidates
Pass 4: Block on Soundex(last name) + birth year → 150K candidates
                    ↓
        Union: 380K unique candidates
        (vs 500B without blocking — 1.3 million x reduction)

The Blocking Trade-off

Metric	Aggressive Blocking	Conservative Blocking
Candidate pairs	Fewer	More
Speed	Faster	Slower
Recall	Lower (miss some matches)	Higher (catch more matches)
Precision	No effect	No effect

The goal: reduce comparisons by 99%+ while missing fewer than 1% of true matches. Multi-pass blocking with diverse keys achieves this reliably.

Blocking with Kanoniv

Kanoniv automatically generates blocking strategies from your matching rules:

rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0
    # → Auto-generates: block on email
  - name: name_address
    type: composite
    children:
      - type: jaro_winkler
        field: name
        threshold: 0.9
      - type: jaro_winkler
        field: address
        threshold: 0.85
    # → Auto-generates: block on (last name prefix + zip)

You don't need to manually configure blocking. The engine derives blocking keys from the rule definitions and runs multiple passes automatically.

Parallelization

After blocking, comparisons within different blocks are independent — making the problem embarrassingly parallel.

Multi-Threaded Comparison

Kanoniv's Rust engine uses Rayon for data parallelism:

Block A (5K pairs) → Thread 1
Block B (3K pairs) → Thread 2
Block C (8K pairs) → Thread 3
Block D (2K pairs) → Thread 4
...

On an 8-core machine, this gives ~6x speedup for the comparison phase (the bottleneck is typically memory bandwidth, not CPU).

Distributed Processing

For datasets too large for a single machine, distribute blocks across workers:

Coordinator
  ├── Worker 1: Blocks A-M (process + return matches)
  ├── Worker 2: Blocks N-Z (process + return matches)
  ├── Worker 3: Blocks 0-9 (process + return matches)
  └── Merger: Combine matches → cluster → golden records

Kanoniv Cloud handles distribution automatically. The local SDK is single-machine but multi-threaded.

Comparison Budget

A practical way to think about scale: how many comparisons can you afford?

Environment	Comparison Budget	Records (with 99% blocking)
Laptop (8 cores)	~10M/sec	100K records in ~10s
Server (32 cores)	~40M/sec	500K records in ~30s
Small cluster (8 nodes)	~300M/sec	2M records in ~60s
Kanoniv Cloud	Auto-scaled	Unlimited

Incremental Processing

Full reconciliation on every run is wasteful when only a small percentage of records change.

The Incremental Approach

Day 1: Full reconciliation (1M records) → 600K golden records
Day 2: 5K new records → Match against existing 600K entities
Day 3: 3K new + 2K updated → Match new, re-evaluate updated

Incremental processing matches new records against the existing identity graph instead of recomparing all records.

Approach	Comparisons (1M base + 5K new)	Time
Full rerun	~500B (all pairs)	Hours
Incremental	~5M (new vs existing)	Seconds

When to Do Full Reconciliation

Incremental matching can drift over time — a chain of incremental updates may produce different results than a full reconciliation. Best practice:

• Daily: Incremental (new and updated records only)
• Weekly: Full reconciliation to catch drift
• On rule changes: Full reconciliation (rules changed = different matches)

Benchmarks

We benchmarked Kanoniv's local SDK on a standard customer deduplication workload: 3 sources, 5 matching rules (2 exact, 3 fuzzy composite), source priority survivorship.

Hardware: MacBook Pro M2 (8 cores, 16GB RAM)

Records	Candidates (after blocking)	Time	Memory	Golden Records
10K	45K	0.8s	85 MB	7.2K
50K	380K	3.2s	210 MB	34K
100K	1.2M	8.1s	420 MB	68K
250K	5.8M	28s	980 MB	165K
500K	18M	72s	1.8 GB	320K

Hardware: EC2 c6i.4xlarge (16 vCPUs, 32GB RAM)

Records	Candidates (after blocking)	Time	Memory	Golden Records
100K	1.2M	4.5s	420 MB	68K
500K	18M	38s	1.8 GB	320K
1M	62M	95s	3.4 GB	612K
2M	210M	340s	6.8 GB	1.18M

Key Observations

1. Blocking is everything. 1M records produce 62M candidates instead of 500B — a 8,000x reduction. Without blocking, the 1M case would take days instead of 95 seconds.

2. Memory scales linearly. ~3.4 MB per 1K records. This is because the Rust engine stores records as compact structs, not Python dicts.

3. Time scales slightly above linearly. O(n * b * log(n)) due to blocking overhead and clustering. In practice, 2x records ≈ 2.5-3x time.

4. Merge rate is consistent. ~60-65% merge rate across dataset sizes, suggesting the synthetic data has consistent duplicate patterns.

Architectural Patterns at Scale

Pattern 1: Warehouse-Native Batch

Snowflake/BigQuery → Extract → Kanoniv → Load golden records → Warehouse

Best for: Data warehouse teams running nightly reconciliation. Golden records live in the warehouse alongside source data.

Pattern 2: Stream + Batch Hybrid

Kafka → Kanoniv Cloud API → Identity Graph (real-time)
                   +
Airflow → Kanoniv SDK → Full reconciliation (nightly)

Best for: Applications needing both real-time resolution and periodic full reconciliation.

Pattern 3: Embedded in Application

User signs up → App calls Kanoniv → "Is this an existing customer?"
                                  → If yes: merge
                                  → If no: create new entity

Best for: SaaS products with built-in deduplication (CRM, support tools, marketing platforms).

Practical Advice

Start Small, Scale Up

Don't architect for 100M records on day one. Start with:

1. 100K records locally — validate your spec, tune thresholds
2. 1M records on a server — test blocking effectiveness
3. 10M+ in the cloud — when you've proven the matching logic works

Invest in Blocking, Not Hardware

If reconciliation is slow, improve blocking before adding CPUs. A better blocking strategy (one that reduces candidates by 10x) is equivalent to a 10x hardware upgrade — and it's free.

Monitor Block Size Distribution

If one block contains 50% of your candidates, that block is too large. Common culprits:

• Blocking on city name (everyone in "New York" is in one block)
• Blocking on common email domain (@gmail.com)
• Missing values creating a "null" block

Fix: use compound blocking keys or exclude common values.

Profile Before Optimizing

Use the reconciliation diagnostics to understand where time is spent:

result = reconcile(sources, spec)
print(f"Normalization: {result.timing.normalize_ms}ms")
print(f"Blocking: {result.timing.block_ms}ms")
print(f"Comparison: {result.timing.compare_ms}ms")
print(f"Clustering: {result.timing.cluster_ms}ms")
print(f"Survivorship: {result.timing.survive_ms}ms")
print(f"Total candidates: {result.total_candidates}")
print(f"Largest block: {result.largest_block_size}")

Comparison is almost always the bottleneck. If blocking or clustering is slow, something is wrong with your configuration.