Matching Rules

Configure how Kanoniv determines whether two records represent the same entity.

How Matching Works

For each pair of candidate records, Kanoniv:

Applies all matching rules defined in your spec
Computes a weighted confidence score
Compares against thresholds to decide Merge / Review / Reject

Score = Σ(rule_score × rule_weight) / Σ(rule_weight)

Rule Types

Exact Match

Binary match on a specific field. Either matches (score = 1.0) or doesn't (score = 0.0). The engine normalizes whitespace and performs case-insensitive comparison by default.

yaml

rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0

Best for: Identifiers that should match exactly — emails, phone numbers, account IDs, domains, tax IDs.

Fuzzy Match (Similarity)

Similarity scoring using string distance algorithms. Returns a score between 0.0 and 1.0 based on how similar two strings are. Records must score above the threshold to count as a match.

yaml

rules:
  - name: name_fuzzy
    type: similarity
    field: name
    weight: 0.3
    algorithm: jaro_winkler
    threshold: 0.85

Best for: Names, addresses, company names — fields where typos, abbreviations, and formatting differences are common.

See Similarity Algorithms below for a detailed breakdown of each algorithm and when to use it.

Range Match

Matches numeric fields within a tolerance. If the absolute difference between two values is within the tolerance, the match scores 1.0.

yaml

rules:
  - name: revenue_range
    type: range
    field: annual_revenue
    tolerance: 10000
    weight: 0.2

Best for: Financial figures, employee counts, dates (as timestamps), or any numeric field where slight differences across sources are expected.

Example: Company A reports revenue of $1,200,000 in the CRM, and $1,195,000 in the billing system. With tolerance: 10000, these match (difference of $5,000 is within tolerance).

Composite Match

Combines multiple rules with AND/OR logic. The composite score is the average of all child rule scores.

yaml

rules:
  - name: company_match
    type: composite
    operator: and    # all children must match
    weight: 0.3
    children:
      - type: similarity
        field: company
        algorithm: jaro_winkler
        threshold: 0.8
      - type: exact
        field: domain

and operator: All child rules must match. Use when you need multiple signals to confirm an identity (e.g., fuzzy name AND exact domain).

or operator: Any child rule matching is sufficient. Use when records might match on different fields depending on the source (e.g., exact email OR exact phone).

yaml

rules:
  - name: contact_match
    type: composite
    operator: or     # any child matching is enough
    weight: 0.5
    children:
      - type: exact
        field: email
      - type: exact
        field: phone
      - type: similarity
        field: name
        algorithm: jaro_winkler
        threshold: 0.9

Best for: Multi-signal matching strategies where you want to express "match on X AND Y" or "match on X OR Y" as a single weighted rule.

Similarity Algorithms

When using type: similarity, the algorithm field determines how string similarity is computed. Kanoniv ships with 6 algorithms — each optimized for a different type of data:

Algorithm	Type	Best For	Scoring
`jaro_winkler`	String distance	Person names, company names	0.0 – 1.0 (continuous)
`levenshtein`	Edit distance	Emails, phones, addresses	0.0 – 1.0 (continuous)
`soundex`	Phonetic	English name variations	Binary (1.0 or 0.0)
`metaphone`	Phonetic	Multi-cultural name variations	Tiered (1.0 / 0.5 / 0.0)
`cosine`	Structural	Company names with word reordering	0.0 – 1.0 (continuous)
`haversine`	Geographic	Location/coordinate matching	1.0 → 0.0 (distance decay)

Jaro-Winkler

yaml

algorithm: jaro_winkler

Measures character-level similarity with a bonus for matching prefixes. Scores range from 0.0 (completely different) to 1.0 (identical).

Pair	Score	Why
John Doe / Jonathan Doe	0.84	Shared prefix "Jo", partial overlap
John Doe / John Doe	1.0	Identical
John Doe / Jane Smith	0.46	Very different
Acme Corp / Acme Corporation	0.91	Shared prefix, similar structure

Best for: Person names, company names — fields where the beginning of the string is most distinctive and transpositions are common ("Jonh" vs "John").

Recommended threshold: 0.85 – 0.92

Levenshtein

yaml

algorithm: levenshtein

Counts the minimum number of single-character edits (insertions, deletions, substitutions) needed to transform one string into another. Kanoniv uses normalized Levenshtein, which divides the edit distance by the length of the longer string, giving a score from 0.0 to 1.0.

Pair	Score	Why
[email protected] / [email protected]	0.92	1 deletion out of 13 chars
555-0101 / 555-0102	0.88	1 substitution out of 8 chars
Robert / Bob	0.17	Very different strings
Acme Corp / Acme Corp.	0.95	1 insertion (period)

Best for: Emails, phone numbers, addresses — fields where single-character typos are the primary source of variation. Works well when strings are roughly the same length.

Recommended threshold: 0.80 – 0.90

Jaro-Winkler vs Levenshtein: Quick Guide

Scenario	Recommended	Why
Person names	`jaro_winkler`	Handles transpositions, prefix bonus helps with nicknames
Company names	`jaro_winkler`	"Acme Corp" vs "Acme Corporation" scores higher
Email addresses	`levenshtein`	Single-char typos are the main variation
Phone numbers	`levenshtein`	Digit transpositions/typos
Street addresses	`levenshtein`	Character-level edits (abbreviations, typos)
Short strings (< 5 chars)	`levenshtein`	Jaro-Winkler's prefix bonus can over-score short matches

Soundex

yaml

algorithm: soundex

Encodes strings into a phonetic representation based on how they sound in English. Two strings that sound alike produce the same Soundex code, regardless of spelling. Scoring is binary — 1.0 if codes match, 0.0 if they don't.

Pair	Soundex Code	Match?
John / Jon	J500 / J500	Yes
Smith / Smyth	S530 / S530	Yes
Robert / Rupert	R163 / R163	Yes
Lee / Li	L000 / L000	Yes

Best for: Person names where phonetic similarity matters more than spelling — immigrant name variations, transliterations, informal nicknames. Particularly useful for English-language names.

Limitations: Only considers the first letter and three consonant codes. Poor for non-English names. "Schmidt" and "Smith" do NOT match (different first letter).

Recommended threshold: 0.5 – 1.0 (binary scoring, so threshold mainly controls whether partial matches via composite rules count)

Metaphone

yaml

algorithm: metaphone

A more advanced phonetic algorithm than Soundex. Uses Double Metaphone to generate two possible phonetic encodings per string, catching more variations. Scoring is tiered: 1.0 if primary codes match, 0.5 if one string's alternate code matches the other's primary code, 0.0 otherwise.

Pair	Match?	Why
John / Jon	Yes (1.0)	Same primary code
Schmidt / Smith	Yes (1.0)	Double Metaphone catches this (Soundex doesn't)
Catherine / Katherine	Yes (1.0)	Same phonetic representation
Sean / Shaun	Yes (1.0)	Both map to the same code

Best for: Cases where Soundex isn't powerful enough — multi-cultural names, names with silent letters, or names where the first letter differs but the pronunciation is the same.

Advantages over Soundex: Handles more phonetic patterns, supports non-English origins better, produces two codes (primary + alternate) for ambiguous pronunciations. The tiered scoring (1.0 for primary match, 0.5 for alternate match) gives more nuance than Soundex's binary output.

Recommended threshold: 0.5 (captures both primary and alternate matches) or 0.9 (primary matches only)

Cosine

yaml

algorithm: cosine

Converts strings into character bigram vectors and computes the cosine of the angle between them. Measures structural similarity independent of word order. Scores range from 0.0 to 1.0.

Pair	Approximate Score	Why
Acme Corporation / Corporation Acme	~0.95	Same bigrams, different order
Data Analytics Inc / Analytics Data Inc	~0.92	Word reordering
John Doe / Doe, John	~0.88	Same tokens rearranged

Best for: Company names and addresses where word order varies across sources — "Acme Corporation" vs "Corporation, Acme" or "123 Main St, Suite 200" vs "Suite 200, 123 Main St". Unlike Jaro-Winkler and Levenshtein, cosine similarity is order-independent.

Recommended threshold: 0.75 – 0.90

Haversine

yaml

algorithm: haversine

Computes the great-circle distance between two geographic coordinates (latitude/longitude). The field value should be in "lat,lon" format (e.g., "40.7128,-74.0060"). Scoring: 1.0 if distance is within the threshold (in km), linear decay to 0.0 at 2x the threshold.

Location A	Location B	Distance	Match (50km threshold)?
40.7128, -74.0060 (NYC)	40.7580, -73.9855 (Midtown)	5.1 km	Yes (score: 1.0)
37.7749, -122.4194 (SF)	37.3382, -121.8863 (San Jose)	48.5 km	Yes (score: 1.0)
37.7749, -122.4194 (SF)	36.7783, -119.4179 (Fresno)	270 km	No (score: 0.0)

Best for: Matching business locations, store addresses, or any entity with geographic coordinates. Useful when two sources have the same business at slightly different lat/lon values (geocoding differences).

Threshold: Set in km. E.g., threshold: 50 means locations within 50km score 1.0, locations between 50–100km get partial scores, locations beyond 100km score 0.0.

yaml

rules:
  - name: location_proximity
    type: similarity
    field: coordinates
    algorithm: haversine
    threshold: 50       # km
    weight: 0.2

Algorithm Selection Quick Reference

Data Type	Recommended Algorithm	Why
Emails with typos	`levenshtein`	Jaro-Winkler over-weights prefix; Levenshtein handles single-char edits better
Person names (Western)	`jaro_winkler`	Handles transpositions and prefix-matching nicknames
Person names (international)	`metaphone`	Soundex only handles English phonetics
Company names	`cosine`	Order-independent — handles "Corp Acme" vs "Acme Corp"
Phones (format variation)	`exact` + `normalizer: phone`	Normalizer strips formatting to E.164; fuzzy matching on digits adds noise
Addresses	`levenshtein` + composite	Combine with exact zip code for corroboration
Domains / URLs	`exact` + `normalizer: domain`	Normalizer strips protocol/www; fuzzy is too permissive

Pre-Match Normalization

Normalizers transform field values before comparison, not during blocking. This is useful when two values represent the same thing but are formatted differently — (555) 123-4567 vs +15551234567, or [email protected] vs [email protected].

Available Normalizers

Name	What It Does	Example
`email`	Lowercases, strips Gmail dots and `+` aliases	`[email protected]` → `[email protected]`
`phone`	Strips formatting, converts to E.164	`(555) 123-4567` → `+15551234567`
`name`	NFC Unicode normalization, lowercase, collapse whitespace	`José García` → `josé garcía`
`nickname`	Everything `name` does + maps common nicknames to canonical form	`Bob` → `robert`, `Liz` → `elizabeth`
`domain`	Strips protocol, `www.`, and trailing path	`https://www.acme.com/about` → `acme.com`
`generic`	Trim whitespace + lowercase	`ACME Corp` → `acme corp`

Usage

Add normalizer to any rule or Fellegi-Sunter field. The normalizer runs on both values before the comparison function:

yaml

rules:
  - name: phone_exact
    type: exact
    field: phone
    weight: 1.0
    options:
      normalizer: phone       # E.164 normalization before comparing

  - name: email_similarity
    type: similarity
    field: email
    algorithm: levenshtein
    threshold: 0.95
    weight: 0.8
    normalizer: email          # Strip aliases before comparing

Normalizers also work on Fellegi-Sunter fields — the normalizer runs before both EM training and scoring:

yaml

decision:
  scoring:
    strategy: fellegi_sunter
    fields:
      - name: email
        comparator: exact
        normalizer: email        # Gmail dots, plus-addressing
        m_probability: 0.95
        u_probability: 0.001
      - name: first_name
        comparator: jaro_winkler
        normalizer: nickname     # Bob → robert before fuzzy compare
        m_probability: 0.85
        u_probability: 0.05

When to Use Normalizers vs Similarity

Scenario	Use
Same value, different formatting	`exact` + normalizer (deterministic, fast)
Similar but not identical values	`similarity` algorithm (handles typos)
Both formatting AND typos	`similarity` + normalizer (normalize first, then fuzzy compare)

Normalizers are deterministic and fast. Prefer exact + normalizer over similarity when the variation is purely formatting.

ML / Custom Models :badge[Coming Soon]

For matching scenarios where built-in algorithms aren't enough — deduplicating unstructured product descriptions, matching addresses across languages, or handling industry-specific identifiers — you can plug in your own machine learning model.

How It Works

The ML rule type lets you point the engine at an external model. You declare which fields to extract as features, the model endpoint, and a confidence threshold. During matching, the engine sends each candidate pair's features to your model and uses the returned score like any other rule.

yaml

rules:
  - name: address_ml
    type: ml
    model: "https://ml.internal/models/address-match"
    features: [street, city, state, zip, country]
    threshold: 0.85
    weight: 0.4

Example: Custom Name Matching with a Hosted Model

Suppose you have a fine-tuned model that handles multi-lingual name matching (e.g., "Muhammad" / "Mohammed" / "Mohamed" or "李明" / "Li Ming"):

yaml

rules:
  # Standard rules handle the easy cases
  - name: email_exact
    type: exact
    field: email
    weight: 0.4

  # Your custom model handles the hard cases
  - name: name_ml
    type: ml
    model: "https://ml.internal/models/multilingual-name-v2"
    features: [name, name_alt, locale]
    threshold: 0.80
    weight: 0.3

  - name: phone_exact
    type: exact
    field: phone
    weight: 0.3

The engine sends a request to your model for each candidate pair:

json

{
  "entity_a": { "name": "李明", "name_alt": "Li Ming", "locale": "zh-CN" },
  "entity_b": { "name": "Ming Li", "name_alt": "", "locale": "en-US" }
}

Your model returns a confidence score (e.g., 0.92), which the engine uses like any other rule score in the weighted average.

Example: Product Deduplication

Match products across catalogs where descriptions vary wildly:

yaml

rules:
  - name: sku_exact
    type: exact
    field: sku
    weight: 0.5

  - name: product_embeddings
    type: ml
    model: "https://ml.internal/models/product-similarity"
    features: [product_name, description, category, brand]
    threshold: 0.75
    weight: 0.5

This lets you combine deterministic rules (exact SKU match) with ML-powered semantic similarity on unstructured text — in the same spec.

Model Requirements

Your model endpoint must:

Accept POST requests with a JSON body containing entity_a and entity_b feature maps
Return a JSON response with a score field (float between 0.0 and 1.0)
Respond within 500ms per pair (the engine batches requests where possible)

INFO

The ML rule type is defined in the spec schema and wired through the engine compiler, but model invocation is not yet connected. Coming in a future release. In the meantime, use composite rules with multiple built-in algorithms to approximate complex matching logic.

Thresholds

Configure decision boundaries in the decision section of your spec:

yaml

decision:
  thresholds:
    match: 0.9
    review: 0.7

Threshold	Score Range	Decision
Above `match`	>= 0.9	Auto-merge into the same canonical entity
Between	0.7 – 0.89	Flagged for manual review
Below `review`	< 0.7	No match — records stay separate

Tuning tips:

Start with match: 0.9 and review: 0.7 — these work well for most datasets
If you're getting false positives (wrong merges), raise the match threshold
If you're getting false negatives (missed merges), lower the match threshold or add more rules
The review queue catches edge cases — use it to iteratively improve your thresholds

Full Spec Example

Rules are defined declaratively in your YAML spec — the single source of truth for matching configuration:

yaml

api_version: kanoniv/v1
identity_version: "1.0"

entity:
  name: customer

sources:
  crm:
    adapter: csv
    location: crm.csv
    primary_key: id
    schema:
      email: { type: string, pii: true }
      name: { type: string }
      phone: { type: string, pii: true }
      company: { type: string }
      annual_revenue: { type: string }

  billing:
    adapter: csv
    location: billing.csv
    primary_key: customer_id
    schema:
      email: { type: string, pii: true }
      full_name: { type: string }
      plan: { type: string }

rules:
  - name: email_exact
    type: exact
    field: email
    weight: 0.4

  - name: name_fuzzy
    type: similarity
    field: name
    algorithm: jaro_winkler
    threshold: 0.85
    weight: 0.3

  - name: phone_exact
    type: exact
    field: phone
    weight: 0.2

  - name: revenue_range
    type: range
    field: annual_revenue
    tolerance: 10000
    weight: 0.1

decision:
  thresholds:
    match: 0.80
    review: 0.60

survivorship:
  strategy: source_priority
  source_order: [crm, billing]

The spec is version-controlled, validated in CI, and applied consistently across every reconciliation run.

Validate Rules

Check that your rules are well-formed before running reconciliation:

python

from kanoniv import Spec, validate

spec = Spec.from_file("customer-spec.yaml")
result = validate(spec)

if result:
    print("Spec is valid — rules look good!")
else:
    for error in result.errors:
        print(f"  {error}")

Match Explanations

Every match decision includes an explanation showing exactly which rules fired and their scores:

python

from kanoniv import Spec, Source, reconcile

spec = Spec.from_file("customer-spec.yaml")
sources = [Source.from_csv("crm", "crm.csv"), Source.from_csv("billing", "billing.csv")]
result = reconcile(sources, spec)

for decision in result.decisions:
    print(decision)

json

{
  "left": "crm:1",
  "right": "billing:cus_001",
  "score": 0.87,
  "outcome": "match",
  "rules": [
    { "name": "email_exact", "score": 1.0, "field": "email" },
    { "name": "name_fuzzy", "score": 0.84, "field": "name" },
    { "name": "phone_exact", "score": 1.0, "field": "phone" }
  ]
}

Every merge has a reason code. This is what makes Kanoniv deterministic and auditable — you always know why two records were matched.

Need real-time resolution with match explanations?

Kanoniv Cloud includes the matched rules and confidence score in every resolution response.

Matching Rules ​

How Matching Works ​

Rule Types ​

Exact Match ​

Fuzzy Match (Similarity) ​

Range Match ​

Composite Match ​

Similarity Algorithms ​

Jaro-Winkler ​

Levenshtein ​

Jaro-Winkler vs Levenshtein: Quick Guide ​

Soundex ​

Metaphone ​

Cosine ​

Haversine ​

Algorithm Selection Quick Reference ​

Pre-Match Normalization ​

Available Normalizers ​

Usage ​

When to Use Normalizers vs Similarity ​

ML / Custom Models :badge[Coming Soon] ​

How It Works ​

Example: Custom Name Matching with a Hosted Model ​

Example: Product Deduplication ​

Model Requirements ​

Thresholds ​

Full Spec Example ​

Validate Rules ​

Match Explanations ​

Matching Rules

How Matching Works

Rule Types

Exact Match

Fuzzy Match (Similarity)

Range Match

Composite Match

Similarity Algorithms

Jaro-Winkler

Levenshtein

Jaro-Winkler vs Levenshtein: Quick Guide

Soundex

Metaphone

Cosine

Haversine

Algorithm Selection Quick Reference

Pre-Match Normalization

Available Normalizers

Usage

When to Use Normalizers vs Similarity

ML / Custom Models :badge[Coming Soon]

How It Works

Example: Custom Name Matching with a Hosted Model

Example: Product Deduplication

Model Requirements

Thresholds

Full Spec Example

Validate Rules

Match Explanations