What Is Entity Resolution? A Complete Guide for Data Engineers

Published February 2026 · 12 min read

You have customer data in Salesforce, billing data in Stripe, support tickets in Zendesk. Each system has its own IDs, its own formatting, and its own version of every customer. "Robert Smith" in the CRM is "Bob Smith" in billing is "R. Smith" in support. Same person, three records, zero shared identifiers.

Entity resolution is the process of figuring out which records across these systems refer to the same real-world person — and linking them together.

This guide covers entity resolution from a data engineering perspective: what it is, how it works, what tools are available, and how to implement it in production.

The Core Problem

Every organization with multiple data systems has duplicate and fragmented records. The scale of the problem:

The average enterprise uses 976 applications (MuleSoft 2023)
Customer data degrades at 25-30% per year due to moves, name changes, and data entry errors
A typical B2B database has 10-30% duplicate records
Poor data quality costs organizations an average of $12.9 million per year (Gartner)

Entity resolution addresses this by matching records that refer to the same entity (person, company, device) and producing a single canonical version.

How Entity Resolution Works

The standard pipeline has five stages:

1. Normalization

Clean and standardize data for comparison:

"Robert J. Smith Jr."     → "robert smith"
"(555) 123-4567"          → "5551234567"
"[email protected]" → "[email protected]"
"123 Main St., Ste. 4"   → "123 main st ste 4"

2. Blocking

Reduce the comparison space. Without blocking, N records require N*(N-1)/2 comparisons — 500 billion for 1M records. Blocking groups records by shared attributes and only compares within groups.

Multiple blocking passes with different keys (email, phone, last name prefix + zip) catch matches that any single strategy would miss.

3. Matching

Compare candidate pairs field-by-field:

Algorithm	Best For	Example
Exact match	Identifiers (email, SSN)	"[email protected]" = "[email protected]"
Jaro-Winkler	Names	"Robert" ↔ "Rob" = 0.91
Levenshtein	Addresses	"123 Main St" ↔ "123 Main Street" = 0.85
Soundex	Phonetic names	"Smith" ↔ "Smyth" = same code

Field scores are combined into an overall match score using weighted sums.

4. Clustering

Group matched pairs into clusters. If A matches B and B matches C, then {A, B, C} form one entity. Connected components graph algorithms handle this efficiently.

5. Golden Record Creation

Select the best field values from each cluster to produce one canonical record. This is called survivorship — deciding which values survive into the golden record.

Strategy	Logic	Example
Source priority	Prefer higher-priority sources	CRM name over billing name
Most recent	Use latest value	Most recent address
Most complete	Use longest/non-null value	Full name over abbreviation

Matching Approaches

Deterministic (Rule-Based)

Write explicit rules: "if email matches, it's the same person."

Pros: Predictable, explainable, fast, no training data. Cons: Misses fuzzy matches, requires manual rule maintenance. Best for: Clean data with strong identifiers.

Probabilistic (Fellegi-Sunter)

Statistical model that computes match likelihood based on field agreements and disagreements. Rare field agreements (unusual name) provide more evidence than common ones ("John Smith").

Pros: Handles messy data, calibrated confidence scores. Cons: Harder to explain, requires parameter estimation. Best for: Large-scale matching with noisy data.

ML-Based

Train a classifier on labeled pairs. Active learning picks the most informative pairs for labeling — typically 30-50 pairs suffice.

Pros: Captures complex patterns, adapts to data. Cons: Requires training data, less explainable. Best for: Messy data where rules can't capture the pattern.

Hybrid (Recommended)

Most production systems combine approaches:

Deterministic rules for high-confidence matches (exact email, exact phone)
Fuzzy/probabilistic for ambiguous cases (similar names + same city)
Human review for borderline pairs

Tools and Frameworks

The entity resolution ecosystem in 2026:

Tool	Type	Approach	Golden Records	Scale
Kanoniv	Platform (OSS + Cloud)	Declarative YAML rules	Yes	100K+ local, unlimited cloud
Splink	Python library	Probabilistic (Fellegi-Sunter)	No	100M+ (Spark/DuckDB)
Zingg	Spark library	ML + active learning	Enterprise only	Large (Spark)
Dedupe	Python library	ML + active learning	No	Small-medium
Senzing	Commercial engine	Zero-config AI	No	Billions (claimed)
AWS Entity Resolution	Managed service	Rule + ML + Provider	No	Large

Key differentiators:

Golden records: Only Kanoniv includes survivorship out of the box. Others stop at matching.
Local development: Kanoniv and Splink run locally. AWS ER requires an AWS account.
Config format: Kanoniv uses version-controlled YAML. Splink uses Python dicts. AWS ER uses CloudFormation.

Implementation: Step by Step

Step 1: Profile Your Data

Before writing matching rules, understand your data:

python

import pandas as pd

# Load from multiple sources
crm = pd.read_csv("salesforce_contacts.csv")
billing = pd.read_csv("stripe_customers.csv")

# Check completeness
print("CRM completeness:")
print(crm[['name', 'email', 'phone', 'address']].notna().mean())

# Check uniqueness (higher = better discriminator)
print("\nCRM uniqueness:")
print(crm[['name', 'email', 'phone']].nunique() / len(crm))

Fields with high completeness and high uniqueness (email, phone) are your best matching candidates.

Step 2: Define Your Spec

yaml

# customer-spec.yaml
entity:
  name: customer
sources:
  - name: crm
    adapter: csv
    location: salesforce_contacts.csv
    primary_key: sf_id
  - name: billing
    adapter: csv
    location: stripe_customers.csv
    primary_key: cus_id
rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0
  - name: phone_name
    type: composite
    children:
      - type: exact
        field: phone
      - type: jaro_winkler
        field: name
        threshold: 0.88
  - name: name_address
    type: composite
    children:
      - type: jaro_winkler
        field: name
        threshold: 0.9
      - type: jaro_winkler
        field: address
        threshold: 0.85
survivorship:
  strategy: source_priority
  priority: [crm, billing]
decision:
  thresholds:
    match: 0.85
    review: 0.65

Step 3: Validate and Run

python

from kanoniv import Spec, Source, reconcile, validate

# Validate first — catches errors before processing
spec = Spec.from_file("customer-spec.yaml")
validate(spec).raise_on_error()

# Load sources
sources = [
    Source.from_csv("crm", "salesforce_contacts.csv"),
    Source.from_csv("billing", "stripe_customers.csv"),
]

# Run reconciliation
result = reconcile(sources, spec)

print(f"Input records: {result.total_input_records}")
print(f"Golden records: {len(result.golden_records)}")
print(f"Merge rate: {result.merge_rate:.1%}")
print(f"Review pairs: {len(result.review_pairs)}")

Step 4: Validate Results

Spot-check matches. Look for false positives (different people merged) and false negatives (same person in different clusters). Adjust thresholds:

Too many false positives? Raise match_threshold (e.g., 0.85 → 0.90)
Too many false negatives? Lower thresholds or add more matching rules
Giant clusters? Add cluster size limits or stricter rules

Step 5: Operationalize

Run reconciliation on a schedule:

python

# In an Airflow DAG
from airflow.operators.python import PythonOperator

def daily_reconciliation():
    from kanoniv import Spec, Source, reconcile, validate
    spec = Spec.from_file("/opt/specs/customer-spec.yaml")
    validate(spec).raise_on_error()
    sources = [
        Source.from_csv("crm", "/data/daily/crm.csv"),
        Source.from_csv("billing", "/data/daily/billing.csv"),
    ]
    result = reconcile(sources, spec)
    result.to_dataframe().to_sql("golden_customers", engine, if_exists="replace")

reconcile_task = PythonOperator(
    task_id="reconcile_customers",
    python_callable=daily_reconciliation,
)

Common Mistakes

1. Skipping Blocking

Without blocking, entity resolution is O(n^2). Even 100K records produce 5 billion comparisons. Always use blocking — it reduces comparisons by 99%+ with minimal match quality loss.

2. Matching on Name Alone

"John Smith" in Chicago is not the same as "John Smith" in Boston. Always combine name with at least one other field (DOB, address, email domain, phone).

3. Ignoring Survivorship

Matching records is only half the problem. Without survivorship, you have clusters but no usable output. Define which field values win when sources disagree.

4. Setting Thresholds Without Testing

Don't guess at 0.85. Sample 50-100 pairs at different score ranges, manually label them, and set thresholds based on where precision drops.

5. Running Once and Walking Away

Data degrades at 25-30% per year. Entity resolution must run continuously — new records, updated records, changed addresses, new data sources. Automate it.

Term	Meaning
Deduplication	Finding duplicates within a single dataset
Record linkage	Matching records across two datasets
Entity resolution	Umbrella term: dedup + linkage + persistent entity model
Identity resolution	Entity resolution for people/devices (marketing context)
MDM	Broader discipline: entity resolution + governance + stewardship

What Is Entity Resolution? A Complete Guide for Data Engineers ​

The Core Problem ​

How Entity Resolution Works ​

1. Normalization ​

2. Blocking ​

3. Matching ​

4. Clustering ​

5. Golden Record Creation ​

Matching Approaches ​

Deterministic (Rule-Based) ​

Probabilistic (Fellegi-Sunter) ​

ML-Based ​

Hybrid (Recommended) ​

Tools and Frameworks ​

Implementation: Step by Step ​

Step 1: Profile Your Data ​

Step 2: Define Your Spec ​

Step 3: Validate and Run ​

Step 4: Validate Results ​

Step 5: Operationalize ​

Common Mistakes ​

1. Skipping Blocking ​

2. Matching on Name Alone ​

3. Ignoring Survivorship ​

4. Setting Thresholds Without Testing ​

5. Running Once and Walking Away ​

Entity Resolution vs Related Concepts ​

Further Reading ​

What Is Entity Resolution? A Complete Guide for Data Engineers

The Core Problem

How Entity Resolution Works

1. Normalization

2. Blocking

3. Matching

4. Clustering

5. Golden Record Creation

Matching Approaches

Deterministic (Rule-Based)

Probabilistic (Fellegi-Sunter)

ML-Based

Hybrid (Recommended)

Tools and Frameworks

Implementation: Step by Step

Step 1: Profile Your Data

Step 2: Define Your Spec

Step 3: Validate and Run

Step 4: Validate Results

Step 5: Operationalize

Common Mistakes

1. Skipping Blocking

2. Matching on Name Alone

3. Ignoring Survivorship

4. Setting Thresholds Without Testing

5. Running Once and Walking Away

Entity Resolution vs Related Concepts

Further Reading