Deterministic vs Probabilistic Matching

Deterministic matching uses exact rules to link records. Probabilistic matching uses statistical models to estimate the likelihood that two records refer to the same entity. Most production identity resolution systems use both — deterministic rules for high-confidence matches and probabilistic models for ambiguous cases.

This guide explains how each approach works, their trade-offs, and how to choose between them.

Deterministic Matching

Deterministic matching applies explicit, human-defined rules. If the rule conditions are met, the records match. If not, they don't. There is no gray zone.

How It Works

Rule: IF email_a == email_b THEN match
Rule: IF phone_a == phone_b AND last_name_a == last_name_b THEN match
Rule: IF ssn_a == ssn_b THEN match

Each rule is a Boolean expression. Records either satisfy the rule (match) or they don't (no match). Rules are evaluated in priority order — if a high-confidence rule matches, lower-priority rules are skipped.

Example

Record A	Record B	Rule	Result
[email protected]	[email protected]	Email exact	Match
555-123-4567 + Smith	555-123-4567 + Smith	Phone + last name	Match
Robert Smith, Chicago	Bob Smith, Chicago	—	No match (no rule fires)

The third pair is the same person, but deterministic matching misses it because "Robert" ≠ "Bob" and no rule accounts for nicknames.

Strengths

• Predictable: Same input always produces the same output. No model drift, no retraining.
• Explainable: Every match decision can be traced to a specific rule. Auditors and compliance teams can understand exactly why two records were linked.
• Fast: Rule evaluation is computationally cheap — typically microseconds per pair.
• No training data: Rules are written by domain experts, not learned from labeled examples.
• Easy to debug: When a match is wrong, you can identify which rule fired and fix it.

Weaknesses

• Misses fuzzy matches: "Robert" vs "Bob", "123 Main St" vs "123 Main Street", typos, formatting differences — all missed unless you write rules for every variation.
• Rule explosion: As you add rules to cover more cases, the rule set grows complex and hard to maintain.
• Brittle: Rules that work for US data may fail on international data. Rules that work today may fail when data formats change.
• Binary decisions: A record either matches or it doesn't. There's no confidence score to help prioritize human review.

When to Use Deterministic Matching

• Your data has strong identifiers (email, phone, SSN, tax ID) that are reliably present and accurate
• You need 100% explainability for compliance or regulatory reasons
• Your data is relatively clean with consistent formatting
• You're matching within a single domain where you understand the data well
• You need real-time matching with microsecond latency

Probabilistic Matching

Probabilistic matching estimates the probability that two records refer to the same entity based on how their fields compare. The most widely used framework is the Fellegi-Sunter model (1969).

How It Works

The Fellegi-Sunter model computes a match weight for each field comparison based on two probabilities:

• m-probability: The probability that the field values agree given that the records are a true match. For email, this might be 0.95 (most true matches share an email).
• u-probability: The probability that the field values agree by chance given that the records are not a match. For email, this might be 0.001 (very unlikely two random people share an email).

The match weight for a field agreement is:

weight = log2(m / u)

For email: log2(0.95 / 0.001) ≈ 9.9 — a strong signal. For city: log2(0.85 / 0.05) ≈ 4.1 — a moderate signal. For gender: log2(0.98 / 0.5) ≈ 1.0 — a weak signal.

The total match score is the sum of all field weights. If the score exceeds an upper threshold, the pair is a match. If it falls below a lower threshold, it's a non-match. Scores in between are flagged for review.

Example

Field	Record A	Record B	Agreement?	Weight
Email	[email protected]	[email protected]	Yes	+9.9
Name	Robert Smith	Bob Smith	Partial (0.72 Jaro-Winkler)	+2.1
Phone	555-123-4567	(missing)	Missing	0.0
City	Chicago	Chicago	Yes	+4.1
Total				+16.1

With an upper threshold of 12.0 and a lower threshold of 6.0, this pair scores 16.1 → match.

Strengths

• Handles messy data: Fuzzy string comparisons, partial agreements, and missing values are all incorporated naturally.
• Calibrated confidence: The match score reflects actual statistical evidence. A score of 16 is stronger evidence than a score of 8.
• Adapts to data: The m and u probabilities can be estimated from the data itself using the EM algorithm — no labeled training data required.
• Rare values count more: An agreement on an unusual name like "Xiaowei Zheng" provides more evidence than an agreement on "John Smith", because the u-probability is lower for rare values.
• Gray zone handling: Pairs that fall between thresholds are flagged for human review rather than forced into a binary decision.

Weaknesses

• Harder to explain: "The log-likelihood ratio exceeded the threshold" is harder for stakeholders to understand than "the emails matched."
• Parameter sensitivity: The m and u probabilities, the thresholds, and the comparison functions all affect results. Bad parameters produce bad matches.
• Computational cost: More expensive than deterministic rules — typically milliseconds per pair rather than microseconds.
• Requires blocking: Without blocking, the comparison space is quadratic. Probabilistic matching is always paired with a blocking strategy.

When to Use Probabilistic Matching

• Your data has inconsistencies, typos, and missing values that deterministic rules can't handle
• You need confidence scores to prioritize human review
• You want to discover non-obvious matches that rule-based systems miss
• You're matching across diverse data sources with different formats and quality levels
• You need to tune precision vs recall with thresholds rather than rewriting rules

Machine Learning Approaches

Beyond Fellegi-Sunter, modern systems use supervised ML for matching:

Supervised Classification

Train a classifier (logistic regression, random forest, gradient boosting) on labeled pairs:

Feature	Match Pair	Non-Match Pair
Name Jaro-Winkler	0.92	0.45
Email exact	1.0	0.0
Phone Levenshtein	0.95	0.30
Address cosine	0.88	0.12
Label	Match	Non-match

Pros: Can capture non-linear interactions between features. A name mismatch might be acceptable if email and phone both match.

Cons: Requires labeled training data. Model is a black box unless you use interpretable models.

Active Learning

Instead of labeling thousands of pairs, the system picks the most informative pairs for a human to label — typically those near the decision boundary. With 30-50 labeled pairs, an active learning system can achieve performance comparable to a model trained on thousands.

Tools like Dedupe and Zingg use this approach.

Deep Learning

Neural networks that learn record representations (embeddings) directly from raw data. Two records are similar if their embeddings are close in vector space.

Pros: Can handle unstructured data, learn complex patterns without manual feature engineering.

Cons: Requires large training datasets, expensive to train, and difficult to explain.

Head-to-Head Comparison

Dimension	Deterministic	Probabilistic	ML-Based
Accuracy on clean data	Excellent	Good	Good
Accuracy on messy data	Poor	Good	Very good
Explainability	Excellent	Good	Poor-Moderate
Setup effort	Low (write rules)	Medium (configure comparisons)	High (label data, train model)
Maintenance	Manual rule updates	Re-estimate parameters	Retrain model
Latency	Microseconds	Milliseconds	Milliseconds-seconds
Training data	None	None (EM) or optional	Required (30-50 min for active learning)
False positive control	Easy (tighten rules)	Good (raise threshold)	Good (raise threshold)
False negative control	Hard (must add rules)	Good (lower threshold)	Good (lower threshold)
Compliance/audit	Easy	Moderate	Hard

The Hybrid Approach

Most production systems combine deterministic and probabilistic matching in a tiered architecture:

Tier 1: Deterministic rules (high confidence)
  ├── Email exact match → MATCH (confidence: 0.99)
  ├── Phone + last name exact → MATCH (confidence: 0.95)
  └── SSN exact → MATCH (confidence: 0.99)

Tier 2: Probabilistic/fuzzy matching (medium confidence)
  ├── Name Jaro-Winkler > 0.9 + Address cosine > 0.85 → MATCH (confidence: 0.88)
  ├── Name Jaro-Winkler > 0.85 + City exact → REVIEW (confidence: 0.72)
  └── Below thresholds → NO MATCH

Tier 3: Human review
  └── Review queue for borderline cases

This tiered approach gives you:

• Speed: Most matches are resolved by fast deterministic rules in Tier 1
• Coverage: Probabilistic matching in Tier 2 catches what rules miss
• Quality: Human review in Tier 3 handles the hardest cases
• Explainability: Tier 1 matches have clear audit trails; Tier 2 matches have confidence scores

Implementing Both with Kanoniv

Kanoniv's declarative spec supports both deterministic and fuzzy matching rules in a single configuration:

entity:
  name: customer
sources:
  - name: crm
    adapter: csv
    location: contacts.csv
    primary_key: id
  - name: billing
    adapter: csv
    location: stripe.csv
    primary_key: id
rules:
  # Tier 1: Deterministic
  - name: email_exact
    type: exact
    field: email
    weight: 1.0
  - name: phone_lastname
    type: composite
    children:
      - type: exact
        field: phone
      - type: exact
        field: last_name
  # Tier 2: Fuzzy/probabilistic
  - name: name_address_fuzzy
    type: composite
    children:
      - type: jaro_winkler
        field: name
        threshold: 0.9
      - type: jaro_winkler
        field: address
        threshold: 0.85
survivorship:
  strategy: source_priority
  priority: [crm, billing]
decision:
  thresholds:
    match: 0.85
    review: 0.65

from kanoniv import Spec, Source, reconcile, validate

spec = Spec.from_file("customer-spec.yaml")
validate(spec).raise_on_error()

sources = [
    Source.from_csv("crm", "contacts.csv"),
    Source.from_csv("billing", "stripe.csv"),
]
result = reconcile(sources, spec)

# High-confidence matches (above match_threshold)
print(f"Matched: {len(result.golden_records)}")

# Borderline cases (between review and match thresholds)
print(f"Review queue: {len(result.review_pairs)}")

Choosing Your Approach

Start with deterministic rules if:

• You have strong identifiers (email, phone, SSN)
• Your data is relatively clean
• You need to ship something quickly

Add probabilistic/fuzzy matching when:

• Deterministic rules miss too many true matches
• Your data has inconsistencies you can't normalize away
• You need confidence scores for a review workflow

Consider ML when:

• You have labeled data or can invest in active learning
• Your matching logic is too complex to express as rules
• You need to handle highly heterogeneous data

Use a hybrid when:

• You're building a production system that needs both precision and recall
• You want deterministic rules for speed and auditability, with fuzzy matching as a fallback

Frequently Asked Questions

Can I start with deterministic and add probabilistic later?

Yes, and this is the recommended approach. Start with deterministic rules on your strongest identifiers. Measure precision and recall. If recall is too low (missing too many matches), add fuzzy matching rules for the fields where variation is highest (names, addresses).

How do I estimate m and u probabilities?

The EM (Expectation-Maximization) algorithm estimates them from unlabeled data. It iteratively estimates which pairs are true matches and which aren't, then updates the probabilities based on those estimates. Most probabilistic matching tools (Splink, Fellegi-Sunter implementations) include built-in EM estimation.

What's the minimum data size for probabilistic matching?

Probabilistic matching works at any scale, but the EM algorithm needs enough data to estimate reliable parameters. In practice, a few thousand records per source is sufficient. With fewer records, consider using manually set m and u probabilities based on domain knowledge.

Is probabilistic matching the same as fuzzy matching?

No. Fuzzy matching refers to comparison functions that tolerate approximate agreements (Jaro-Winkler, Levenshtein). Probabilistic matching is a framework for combining multiple comparison scores into a calibrated match probability. Probabilistic matching typically uses fuzzy comparisons as inputs, but fuzzy comparisons can also be used in deterministic rules (e.g., "if Jaro-Winkler similarity > 0.9, match").