What Is Identity Resolution?

Identity resolution is the process of determining that multiple data records refer to the same real-world entity — a person, company, household, or device — across different systems, formats, and identifiers. It is the foundation of Customer 360 initiatives, data deduplication, and master data management.

In practical terms: when your CRM has "Robert Smith" at "123 Main St" and your billing system has "Bob Smith" at "123 Main Street," identity resolution determines that these records describe the same person and links them together.

Why Identity Resolution Matters

Every organization with more than one data source has an identity problem. The numbers tell the story:

• The average enterprise uses 976 applications (MuleSoft 2023 Connectivity Report), each with its own customer records
• Customer data degrades at 25-30% per year due to moves, name changes, email changes, and data entry errors
• Poor data quality costs organizations an average of $12.9 million per year (Gartner)
• 73% of companies report that data silos are a significant barrier to digital transformation

Without identity resolution, organizations face:

Problem	Business Impact
Duplicate customer records	Inflated customer counts, wasted marketing spend
Fragmented customer views	Poor personalization, repeated outreach
Inconsistent data across teams	Conflicting reports, eroded trust in data
Compliance risk	Inability to fulfill GDPR "right to be forgotten" across all systems
Revenue leakage	Missed cross-sell opportunities, duplicate payments

How Identity Resolution Works

Identity resolution typically follows a four-stage pipeline:

1. Data Collection and Normalization

Raw records are collected from source systems (CRM, billing, marketing, support) and normalized into a common format. Normalization includes:

• Name standardization: "Robert" → "robert", "SMITH" → "smith", "Jr." removed
• Address parsing: "123 Main St, Ste 4" → street: "123 Main St", unit: "4"
• Phone formatting: "(555) 123-4567" → "+15551234567"
• Email normalization: "[email protected]" → "[email protected]"

2. Blocking (Candidate Selection)

Comparing every record to every other record is computationally infeasible at scale. For 1 million records, that would be ~500 billion comparisons. Blocking reduces this to a manageable set by grouping records that share a common attribute:

• Records that share the same email domain
• Records with the same first 3 characters of last name + same zip code
• Records with the same phone number prefix

Effective blocking reduces comparisons by 99%+ while missing fewer than 1% of true matches.

3. Matching (Comparison and Scoring)

Candidate pairs are compared field-by-field using matching algorithms:

Algorithm	What It Does	Best For
Exact match	Checks string equality	Identifiers (SSN, email, phone)
Jaro-Winkler	String similarity weighted toward prefix	Names
Levenshtein	Edit distance (insertions, deletions, substitutions)	Addresses, general strings
Soundex / Metaphone	Phonetic similarity	Names with spelling variations
Cosine similarity	Vector-space similarity	Longer text fields
Haversine	Geographic distance	Latitude/longitude coordinates
Date tolerance	Allows a window around dates	Birth dates, transaction dates

Each comparison produces a score. Scores are combined (typically as a weighted sum) to produce an overall match confidence.

4. Decision and Clustering

Based on the match score, the system makes a decision:

• Match (score above threshold): Records are linked together
• Review (score in gray zone): Flagged for human review
• Reject (score below threshold): Records are kept separate

Matched records are grouped into clusters, where each cluster represents one real-world entity. Some systems go further and produce a golden record — a single canonical record for each cluster with the best available value for each field.

Identity Resolution Techniques

Deterministic Matching

Deterministic matching uses explicit rules: "if email matches exactly, it's the same person." Rules are defined by humans and applied as-is.

Pros:

• 100% predictable — same input always produces same output
• Easy to explain and audit
• No training data needed

Cons:

• Misses matches when data is messy (typos, abbreviations, missing fields)
• Rules must be manually written and maintained
• Rigid — doesn't adapt to data patterns

Best for: High-confidence matching on clean identifiers (email, phone, SSN).

Probabilistic Matching

Probabilistic matching uses statistical models to estimate the likelihood that two records refer to the same entity. The most well-known framework is the Fellegi-Sunter model (1969), which computes match weights based on how much each field agreement or disagreement shifts the probability of a true match.

Pros:

• Handles messy, incomplete data better than deterministic rules
• Produces calibrated confidence scores
• Can discover non-obvious matches

Cons:

• Harder to explain individual decisions
• Requires parameter estimation (EM algorithm or labeled data)
• More complex to implement and debug

Best for: Large-scale record linkage with noisy data, research contexts.

ML-Based Matching

ML-based matching trains a classifier (logistic regression, random forest, or neural network) to predict whether two records are a match. Training data comes from labeled examples, often via active learning where the system picks the most informative pairs for a human to label.

Pros:

• Can capture complex, non-linear patterns
• Adapts to specific data characteristics
• Active learning reduces labeling effort (typically 30-50 labeled pairs)

Cons:

• Requires training data (even if active learning reduces the amount)
• Model is a black box unless interpreted carefully
• May not generalize well to new data distributions

Best for: Messy, heterogeneous data where rules are hard to articulate.

Hybrid Approaches

Most production systems combine techniques:

1. Deterministic rules handle high-confidence matches (exact email, exact phone)
2. Probabilistic or ML models handle the remaining ambiguous cases
3. Human review catches edge cases flagged by the model

This layered approach maximizes precision on easy cases and recall on hard cases.

Key Concepts

Golden Records and Survivorship

A golden record is the single, canonical representation of an entity after identity resolution. Survivorship is the process of choosing which field values make it into the golden record when sources disagree.

Common survivorship strategies:

Strategy	How It Works	Example
Source priority	Prefer values from higher-priority sources	CRM address over billing address
Most recent	Use the most recently updated value	Latest phone number wins
Most complete	Pick the non-null or longest value	Full name over abbreviation
Aggregation	Combine values from multiple sources	Union of all email addresses

Identity Graphs

An identity graph maps the relationships between external identifiers (email, phone, cookie ID, device ID) and canonical entity IDs. It answers the question: "given this identifier, who is this person?"

Identity graphs power real-time use cases like:

• Personalization: "This website visitor with cookie X is the same person as customer Y in our CRM"
• Fraud detection: "This new account shares a phone number with a known fraudulent account"
• Attribution: "This ad click led to this purchase, even though different devices were used"

Entity Resolution vs. Record Linkage vs. Deduplication

These terms are often used interchangeably, but they have distinct meanings:

Term	What It Means
Deduplication	Finding duplicate records within a single dataset
Record linkage	Matching records across two or more datasets
Entity resolution	The umbrella term covering both, plus maintaining a persistent entity model over time
Identity resolution	Entity resolution applied specifically to people, households, or devices (often in marketing/ad-tech context)

Implementing Identity Resolution

Step 1: Define Your Entity Model

What are you resolving? Customers? Patients? Companies? Devices? Define the fields that matter and which sources contribute them.

Step 2: Choose Your Matching Strategy

• If your data has strong identifiers (email, phone, SSN): start with deterministic rules
• If your data is messy with lots of typos: consider fuzzy matching with Jaro-Winkler or Levenshtein
• If you can't articulate rules: try ML-based matching with active learning
• For production systems: use a hybrid approach with deterministic rules first, fuzzy matching second

Step 3: Define Survivorship

Don't stop at matching. Define how to produce golden records:

• Which source is most authoritative for each field?
• How do you handle conflicts?
• How do you handle nulls?

Step 4: Operationalize

Identity resolution isn't a one-time job. New data arrives constantly. You need:

• Incremental matching for new records
• Monitoring for match quality over time
• Audit trails for compliance
• APIs for real-time resolution in applications

Example: Identity Resolution with Kanoniv

Kanoniv takes a declarative approach — you define your entire identity resolution logic in a YAML spec:

entity:
  name: customer
sources:
  - name: crm
    adapter: csv
    location: contacts.csv
    primary_key: id
  - name: billing
    adapter: csv
    location: stripe.csv
    primary_key: id
rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0
  - name: name_fuzzy
    type: jaro_winkler
    field: name
    threshold: 0.9
    weight: 0.8
survivorship:
  strategy: source_priority
  priority: [crm, billing]
decision:
  thresholds:
    match: 0.85

from kanoniv import Spec, Source, reconcile, validate

spec = Spec.from_file("customer-spec.yaml")
validate(spec).raise_on_error()

sources = [
    Source.from_csv("crm", "contacts.csv"),
    Source.from_csv("billing", "stripe.csv"),
]
result = reconcile(sources, spec)
print(f"Golden records: {len(result.golden_records)}")
print(f"Merge rate: {result.merge_rate:.1%}")

The spec is version-controlled, portable between local development and cloud, and serves as living documentation of your matching logic.

Identity Resolution Tools Landscape

The identity resolution market includes open-source libraries, commercial platforms, and cloud services:

Tool	Type	Approach	Golden Records	Real-time
Kanoniv	Platform (OSS + Cloud)	Declarative YAML rules	Yes	Yes
Splink	OSS library	Probabilistic (Fellegi-Sunter)	No	No
Zingg	OSS library (Spark)	ML + active learning	Enterprise only	No
Dedupe	OSS library	ML + active learning	No	No
Senzing	Commercial engine	Principle-based AI	No	Yes
AWS Entity Resolution	Managed service	Rule + ML + Provider	No (needs Glue ETL)	Near real-time
Informatica MDM	Enterprise platform	Rule + ML	Yes	Yes
Amperity	CDP	ML-based	Yes	Yes

For detailed comparisons, see our comparison pages.

Frequently Asked Questions

How is identity resolution different from deduplication?

Deduplication finds duplicates within a single dataset. Identity resolution is broader — it matches records across multiple datasets, maintains a persistent entity model, and typically includes golden record creation and real-time resolution APIs.

What accuracy should I expect?

Well-tuned identity resolution systems typically achieve 95-99% precision (of records matched, what fraction are truly the same entity) and 90-98% recall (of true matches, what fraction did the system find). The exact numbers depend on data quality, matching strategy, and domain.

How much data do I need to get started?

You can start with as few as two CSV files. Identity resolution doesn't require a minimum data volume — it's about connecting records across sources, not processing a minimum amount.

Can identity resolution work in real-time?

Yes. Modern identity resolution platforms like Kanoniv provide sub-millisecond resolution APIs that answer "who is this?" in real-time. This is essential for use cases like fraud detection, personalization, and customer support.

Is identity resolution the same as a Customer Data Platform (CDP)?

No. A CDP is a broader category that includes data collection, audience segmentation, and activation in addition to identity resolution. Identity resolution is one component of a CDP — arguably the most important one, since everything else depends on knowing who your customers are.