Algorithms Overview

Kanoniv resolves identities through a five-stage pipeline. Each stage uses one or more algorithms that you configure in your spec. This section explains how every algorithm works, when to choose it, and how they interact.

The Pipeline

Stage 1: Blocking

Before comparing records, Kanoniv partitions your data into blocks — groups of records that share a blocking key. Only records within the same block are compared, reducing the comparison space from O(N²) to something manageable.

Strategies: composite, single, lsh, ml_predictedTransformations: lowercase, soundex, double_metaphone, first_initial, area_code, last4, trigrams, normalize_phone, normalize_email, normalize_name, normalize_domain

Learn more →

Stage 2: Comparison

For each candidate pair, Kanoniv runs your matching rules. Each rule compares one or more fields using an algorithm and produces a score between 0.0 and 1.0.

Algorithms: exact, jaro_winkler, levenshtein, soundex, metaphone, cosine, haversineOperators: AND, OR (for composite rules)

Learn more →

Stage 3: Scoring

Individual rule scores are combined into a single composite score for each pair. The scoring strategy determines how rule scores are weighted and combined.

Methods: weighted_sum (default), fellegi_sunter, ml_ensemble, custom

Learn more →

Stage 4: Decision

The composite score is compared against thresholds to classify each pair as a match, review candidate, or rejection.

decision:
  thresholds:
    match: 0.9     # score >= 0.9 → auto-merge
    review: 0.7    # 0.7 <= score < 0.9 → human review
    reject: 0.3    # score < 0.3 → no match

Stage 5: Survivorship

Matched records are merged into a golden record — one canonical representation per entity. For each field, a survivorship strategy determines which value wins when sources disagree.

Strategies: source_priority, most_recent, most_complete, aggregate, customAggregate functions: union, intersection, max, min, sum, avg

Learn more →

How Algorithms Compose

Here's a complete spec showing all five stages working together:

api_version: kanoniv/v2
identity_version: "1.0"
entity:
  name: customer

sources:
  crm:
    adapter: postgres
    location: public.contacts
    primary_key: id
    attributes:
      name: full_name
      email: email_address
      phone: phone_number
  billing:
    adapter: csv
    location: stripe_customers.csv
    primary_key: customer_id
    attributes:
      name: name
      email: email
      phone: phone

# Stage 1: Blocking
blocking:
  strategy: composite
  keys:
    - fields: [email]
    - fields: [phone]
      transformations: [normalize_phone, last4]

# Stage 2: Comparison
rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0
  - name: name_fuzzy
    type: similarity
    field: name
    algorithm: jaro_winkler
    threshold: 0.88
    weight: 0.6
  - name: phone_exact
    type: exact
    field: phone
    weight: 0.85

# Stage 3 + 4: Scoring & Decision
decision:
  scoring: weighted_sum
  thresholds:
    match: 0.9
    review: 0.7
    reject: 0.3

# Stage 5: Survivorship
survivorship:
  default: source_priority
  source_ranking: [crm, billing]
  overrides:
    - field: email
      strategy: most_recent
    - field: tags
      strategy: aggregate
      function: union

Quick Reference

Stage	Algorithm	Type	Description
Comparison	exact	Rule	Binary 1.0/0.0 on exact string equality
	jaro_winkler	Rule	Edit distance weighted toward prefix matches (names)
	levenshtein	Rule	Normalized edit distance (addresses, general text)
	soundex	Rule	Phonetic encoding — sounds-alike matching
	metaphone	Rule	Advanced phonetic encoding — handles more languages
	cosine	Rule	Token-overlap similarity (company names, descriptions)
	haversine	Rule	Geographic distance between lat/long coordinates
Blocking	composite	Strategy	Multiple blocking keys combined with OR
	single	Strategy	Single blocking key — strictest partitioning
	lsh	Strategy	MinHash with bands/rows — fuzzy blocking
	ml_predicted	Strategy	ML-predicted blocking keys
Scoring	weighted_sum	Method	Default: sum(score × weight) / sum(weights)
	fellegi_sunter	Method	Probabilistic: log-likelihood ratios with m/u probabilities
	ml_ensemble	Method	Logistic regression with trained coefficients
	custom	Method	User-defined formula with arithmetic operators
Survivorship	source_priority	Strategy	Ranked source list — highest non-null wins
	most_recent	Strategy	Latest timestamp wins
	most_complete	Strategy	Longest / fullest value wins
	aggregate	Strategy	Combine values: union, intersection, max, min, sum, avg

Next Steps

• String Comparators — Learn when to use each comparison algorithm
• Blocking Strategies — Understand the N² problem and how to solve it
• Scoring Strategies — Deep dive into Fellegi-Sunter and EM training
• Survivorship — Build golden records from conflicting sources