Entity Resolution Explained

Entity resolution is the process of identifying records in one or more datasets that refer to the same real-world entity, even when those records use different identifiers, formats, or contain errors. Also known as record linkage, data matching, or deduplication, entity resolution is a fundamental building block of data integration, data quality, and master data management.

This guide explains how entity resolution works at a technical level — the algorithms, the architecture, and the engineering decisions that determine whether your system produces reliable results at scale.

The Problem

Consider three records from three systems:

Source	ID	Name	Email	Phone	Address
CRM	C-101	Robert J. Smith	[email protected]	555-123-4567	123 Main St, Chicago IL
Billing	INV-42	Bob Smith	[email protected]	(555) 123-4567	123 Main Street, Chicago
Support	T-8891	R. Smith	[email protected]		123 Main, Chicago IL 60601

A human can see these are the same person. But there is no shared identifier — the CRM ID, billing ID, and support ticket ID are all different. The name varies ("Robert J. Smith" vs "Bob Smith" vs "R. Smith"). The address is formatted differently. The support record is missing a phone number.

Entity resolution automates this judgment at scale: matching records despite inconsistencies, grouping them into clusters, and (optionally) producing a single canonical record.

The Pipeline

Entity resolution is typically implemented as a pipeline with four stages:

Sources → Normalize → Block → Match → Cluster → Golden Record

Stage 1: Normalization

Raw data is cleaned and standardized to improve comparison accuracy:

• Case normalization: "ROBERT SMITH" → "robert smith"
• Whitespace/punctuation: "123 Main St." → "123 main st"
• Abbreviation expansion: "St" → "Street", "Jr" → ""
• Phone formatting: "(555) 123-4567" → "5551234567"
• Email canonicalization: "[email protected]" → "[email protected]"
• Unicode normalization: "café" → "cafe" (NFD/NFC)

Normalization should be deterministic and reversible — the original values are preserved, and normalized forms are used only for comparison.

Stage 2: Blocking

The core scalability technique. Without blocking, comparing N records requires N*(N-1)/2 comparisons — 500 billion for 1 million records. Blocking reduces this to a manageable number.

How it works: Records are grouped into "blocks" by one or more attributes. Only records within the same block are compared.

Blocking Strategy	Example	Reduction
Exact attribute	Same zip code	~99%
Prefix	First 3 chars of last name	~95%
Phonetic	Same Soundex code	~97%
Token	Shared email domain	~98%
Compound	Same city + first initial	~99.5%

Multiple blocking passes are used to catch matches that would be missed by any single strategy. For example:

• Pass 1: Block on email → catches "[email protected]" matches
• Pass 2: Block on (last name prefix + zip) → catches matches without shared email
• Pass 3: Block on phone number → catches matches with shared phone

The union of all passes forms the candidate set.

Stage 3: Matching

Each candidate pair is compared field-by-field. The comparison functions depend on the field type:

String similarity metrics:

Metric	Formula	Range	Best For
Jaro-Winkler	Weighted character transpositions, prefix bonus	0.0 - 1.0	Names
Levenshtein	Minimum edit operations	0 - ∞ (normalized: 0.0 - 1.0)	Short strings, addresses
Damerau-Levenshtein	Levenshtein + transpositions	0 - ∞	Typo detection
Cosine (TF-IDF)	Vector-space token similarity	0.0 - 1.0	Longer text, addresses
Jaccard	Set intersection / set union of tokens	0.0 - 1.0	Multi-word fields

Phonetic algorithms:

Algorithm	How It Works
Soundex	Encodes consonants into a 4-character code (S530 for "Smith")
Metaphone	Improved phonetic encoding handling more linguistic rules
NYSIIS	New York State Identification system for names

Scoring: Field-level scores are combined into an overall match score. Two common approaches:

1. Weighted sum: Each field gets a weight reflecting its discriminating power. Email exact match might get weight 0.9; city match gets 0.2.

2. Fellegi-Sunter: Each field comparison shifts a log-likelihood ratio. Agreement on a rare value (e.g., an unusual name) provides more evidence than agreement on a common value (e.g., "Smith" in the US).

Stage 4: Clustering

Matched pairs are grouped into clusters using graph algorithms:

• Connected components: If A matches B and B matches C, then {A, B, C} form one cluster. Simple but can create oversized clusters from transitive chains.
• Correlation clustering: Optimizes both within-cluster similarity and between-cluster dissimilarity. More accurate but computationally expensive.
• Thresholded clustering: Only add a record to a cluster if it matches at least one existing member above a threshold.

Stage 5: Golden Record Creation (Survivorship)

Once clusters are identified, the final step is producing a golden record — one canonical record per entity. This requires survivorship rules to resolve conflicts:

Cluster: {CRM record, Billing record, Support record}
                    ↓ Survivorship
Golden Record: Best name from CRM, email from Support, phone from CRM, address from CRM

Not all entity resolution tools include this step. Libraries like Splink and Dedupe stop at clustering. Platforms like Kanoniv include survivorship as a first-class feature.

Matching Approaches Compared

Approach	Configuration	Training Data	Handles Noise	Explainability
Deterministic	Hand-written rules	None	Poor	Excellent
Probabilistic (Fellegi-Sunter)	Comparison functions + EM	None (unsupervised)	Good	Good
ML (supervised)	Feature engineering + classifier	Labeled pairs	Very good	Moderate
ML (active learning)	Feature engineering + classifier	Small set (30-50 pairs)	Very good	Moderate
Deep learning	Embeddings + neural network	Large labeled dataset	Excellent	Poor
Hybrid	Rules + ML fallback	Optional	Very good	Good

Architecture Patterns

Batch Entity Resolution

Data Lake → ETL → Entity Resolution Job → Golden Records → Warehouse

• Runs on a schedule (daily, hourly)
• Processes all records or incremental updates
• Results written to a data warehouse or lake
• Tools: Splink, Zingg, Dedupe, AWS Entity Resolution

Real-Time Entity Resolution

Event → API → Resolution Engine → Response (< 10ms)

• Processes one record at a time as events arrive
• Queries a persistent identity graph
• Returns the resolved entity immediately
• Tools: Kanoniv Cloud, Senzing

Hybrid Architecture

Batch: Nightly reconciliation → Golden Records → Warehouse
Real-time: API requests → Identity Graph → Instant resolution

Most production systems use both:

• Batch for initial load and periodic full reconciliation
• Real-time for incoming records and application queries

Common Pitfalls

1. Over-Matching (False Positives)

Problem: "John Smith in Chicago" matches "John Smith in Boston" — different people.

Solutions:

• Increase match thresholds
• Add discriminating fields (DOB, phone) as required match criteria
• Use compound rules: name + address must both match

2. Under-Matching (False Negatives)

Problem: "Robert Smith" doesn't match "Bob Smith" — same person, missed.

Solutions:

• Add fuzzy matching (Jaro-Winkler for names)
• Add phonetic matching (Soundex)
• Use multiple blocking strategies to increase candidate coverage

3. Chain Linking

Problem: A matches B (genuine), B matches C (genuine), but A and C are different people. Connected components clustering merges them all.

Solutions:

• Use stricter clustering algorithms (correlation clustering)
• Set maximum cluster size limits
• Use review thresholds for borderline matches

4. Data Drift

Problem: People move, change names, change jobs. Matching rules that worked last year fail this year.

Solutions:

• Re-evaluate match quality regularly
• Use temporal decay (recent data is more reliable)
• Monitor match rates and alert on anomalies

Measuring Quality

Entity resolution quality is measured along two axes:

Metric	Definition	Target
Precision	Of records matched, % that are truly the same entity	> 95%
Recall	Of true matches, % that the system found	> 90%
F1 Score	Harmonic mean of precision and recall	> 92%

There is always a precision-recall trade-off:

• Higher thresholds → more precision, less recall (fewer false matches, more missed matches)
• Lower thresholds → more recall, less precision (catches more matches, but some are wrong)

The right balance depends on your use case:

• Fraud detection: Favor recall (don't miss fraudsters)
• Marketing personalization: Favor precision (don't merge different people)
• Regulatory compliance: Favor recall (must find all records for a data subject)

Entity Resolution with Kanoniv

Kanoniv provides a complete entity resolution pipeline in a declarative YAML spec:

entity:
  name: customer
sources:
  - name: crm
    adapter: csv
    location: contacts.csv
    primary_key: id
  - name: billing
    adapter: csv
    location: stripe.csv
    primary_key: id
rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0
  - name: name_address
    type: composite
    children:
      - type: jaro_winkler
        field: name
        threshold: 0.9
      - type: jaro_winkler
        field: address
        threshold: 0.85
survivorship:
  strategy: source_priority
  priority: [crm, billing]
decision:
  thresholds:
    match: 0.85
    review: 0.65

This spec covers the entire pipeline: sources, normalization, matching rules, survivorship, and decision thresholds. It runs identically on a laptop during development and in Kanoniv Cloud for production.

Frequently Asked Questions

What is the difference between entity resolution and MDM?

Entity resolution is the matching and linking component. Master Data Management (MDM) is a broader discipline that includes data governance, stewardship workflows, data quality rules, and golden record management. Entity resolution is a building block of MDM, not a replacement for it.

Can entity resolution handle billions of records?

Yes, with the right architecture. Blocking reduces the comparison space, and distributed computing (Spark, cloud services) parallelizes the work. Senzing claims billions of records on 1000-node clusters. AWS Entity Resolution handles large datasets as a managed service. Splink handles 100M+ records on Spark.

Does entity resolution require machine learning?

No. Deterministic matching with explicit rules works well for many use cases, especially when data has strong identifiers (email, phone, SSN). ML-based approaches are most valuable when data is messy and rules are hard to articulate. Many production systems use deterministic rules for 80% of matches and ML for the remaining 20%.

How often should entity resolution run?

It depends on data velocity. Batch systems run daily or hourly. Real-time systems process records as they arrive. For most B2B data teams, a daily batch reconciliation with an incremental real-time API for new records is a good starting architecture.