Blocking Strategies

Blocking is the most important performance optimization in identity resolution. Without it, comparing every record to every other record is computationally infeasible at scale.

The N² Problem

Comparing every record pair grows quadratically:

Records	Pairs	At 10K pairs/sec
1,000	499,500	50 seconds
10,000	49,995,000	83 minutes
100,000	4,999,950,000	5.8 days
1,000,000	499,999,500,000	1.6 years

Blocking solves this by partitioning records into groups that share a blocking key. Only records within the same block are compared. Good blocking reduces pairs by 99%+ while missing less than 1% of true matches.

How Blocking Works in Kanoniv

blocking:
  strategy: composite
  keys:
    - fields: [email]
    - fields: [phone]
      transformations: [normalize_phone, last4]
    - fields: [last_name]
      transformations: [soundex]

Each blocking key defines a partition. Records with the same blocking key value are placed in the same block. With composite strategy, Kanoniv evaluates all keys and takes the union of candidate pairs — a pair only needs to share one blocking key to be compared.

Field Blocking

The simplest strategy: partition records by the exact value of one or more fields.

How It Works

Records are grouped by the blocking key value. Only records with identical key values are compared. With multiple fields in one key, the values are concatenated.

Example

Blocking on [email] with 100K records:

Block Key	Records	Pairs in Block
[email protected]	3	3
[email protected]	2	1
[email protected]	5	10
...	...	...
Total	100,000	~12,000

From 5 billion possible pairs down to ~12K — a 99.9998% reduction.

YAML Config

Single key (strictest):

blocking:
  strategy: single
  keys:
    - fields: [email]

Composite keys (recommended):

blocking:
  strategy: composite
  keys:
    - fields: [email]
    - fields: [phone]
    - fields: [last_name, zip_code]

When to Use

• You have a high-quality identifier (email, phone, SSN)
• Data is well-structured with low null rates
• You need maximum performance

Limitations

If [email protected] in one source and [email protected] in another, they land in different blocks and are never compared. Combine with fuzzy blocking keys to catch these.

Domain Blocking

Extract the domain portion of an email or URL and block on that.

How It Works

The normalize_domain transformation extracts the domain:

• [email protected] → acme.com
• https://www.acme.com/about → acme.com
• [email protected] → acme.com

Records from the same domain are compared. This creates larger blocks than email blocking but catches records where the username differs.

Example

Source	Email	Domain Key
CRM	[email protected]	acme.com
Stripe	[email protected]	acme.com
Zendesk	[email protected]	acme.com
CRM	[email protected]	beta.io

All three acme.com records are compared to each other, even though their emails differ.

YAML Config

blocking:
  strategy: composite
  keys:
    - fields: [email]
      transformations: [normalize_domain]
    - fields: [website]
      transformations: [normalize_domain]

When to Use

• B2B matching where company domain is a strong signal
• Matching contacts across CRM, billing, and support systems
• Data where email usernames vary but domains are consistent

Sorted Neighborhood

A sliding-window approach that catches near-misses in sorted order.

How It Works

1. Records are sorted by a blocking key (e.g., normalized name)
2. A window of size w slides across the sorted list
3. All records within the window are compared

Because similar strings sort close together, typos and minor variations end up in the same window. A window size of 5–10 catches most near-misses without exploding the comparison count.

Example

Sorted by normalize_name(last_name):

Position	Last Name	Normalized
1	Smith	smith
2	Smit	smit
3	Smyth	smyth
4	Snider	snider
5	Snow	snow

With window size 3, positions 1-3 are compared (Smith/Smit/Smyth), then 2-4, then 3-5. Smith and Smyth are compared even though they wouldn't share an exact blocking key.

YAML Config

blocking:
  strategy: composite
  keys:
    - fields: [last_name]
      transformations: [normalize_name]

When to Use

• Name-based matching where typos are common
• Data without strong identifiers (no email, no phone)
• When exact blocking misses too many true matches

Phonetic Blocking

Apply a phonetic transformation to the blocking key so that similar-sounding values land in the same block.

How It Works

Instead of blocking on the raw field value, Kanoniv applies a transformation that maps phonetically similar strings to the same key:

• Soundex: Smith → S530, Smyth → S530 (same block)
• Double Metaphone: Catherine → K0RN, Katherine → K0RN (same block)

YAML Config

blocking:
  strategy: composite
  keys:
    - fields: [last_name]
      transformations: [soundex]
    - fields: [first_name]
      transformations: [double_metaphone]

When to Use

• Name-heavy datasets with spelling variants
• Healthcare patient matching (common: Stephen/Steven, Catherine/Katherine)
• Combine with exact email blocking for balanced recall/performance

Blocking Transformations

Transformations modify field values before computing the blocking key. Apply them to any blocking key:

Transformation	Input	Output	Use Case
lowercase	Jon Smith	jon smith	Case-insensitive blocking
soundex	Smith	S530	Phonetic name blocking
double_metaphone	Katherine	K0RN	Advanced phonetic blocking
first_initial	Jonathan	j	Coarse name partitioning
area_code	+1-555-010-0100	555	Region-based phone blocking
last4	+1-555-010-0100	0100	Partial phone matching
trigrams	Smith	{smi, mit, ith}	Character n-gram blocking
normalize_phone	(555) 010-0100	+15550100100	Consistent phone format
normalize_email	[email protected]	[email protected]	Consistent email format
normalize_name	Dr. John Smith Jr.	john smith	Strip titles and suffixes
normalize_domain	https://www.acme.com	acme.com	Domain extraction

Chaining Transformations

Transformations are applied in order. Chain them for compound keys:

blocking:
  strategy: composite
  keys:
    - fields: [phone]
      transformations: [normalize_phone, area_code]   # (555) 010-0100 → +15550100100 → 555
    - fields: [last_name]
      transformations: [normalize_name, soundex]       # Dr. Smith Jr. → smith → S530
    - fields: [email]
      transformations: [normalize_email]               # [email protected] → [email protected]

LSH (Locality-Sensitive Hashing)

For fuzzy blocking at scale, LSH uses MinHash to probabilistically group similar records.

How It Works

1. Each record's blocking fields are tokenized into character n-grams (shingles)
2. A MinHash signature is computed — a fixed-size sketch that preserves Jaccard similarity
3. The signature is divided into bands of rows each
4. Records that hash to the same bucket in any band become candidates

The bands/rows parameters control the trade-off:

• More bands, fewer rows: lower similarity threshold — more candidates, higher recall, slower
• Fewer bands, more rows: higher similarity threshold — fewer candidates, lower recall, faster

YAML Config

blocking:
  strategy: lsh
  keys:
    - fields: [name, address]

When to Use

• Very large datasets (1M+ records) where exact blocking misses too many matches
• Data with no strong identifiers
• When recall is more important than speed

Trade-offs

LSH generates more candidate pairs than exact blocking, so comparison is slower. But it catches fuzzy matches that exact blocking misses entirely. Use it as a supplementary strategy alongside exact blocking:

blocking:
  strategy: composite
  keys:
    - fields: [email]                              # Exact — fast, high precision
    - fields: [phone]
      transformations: [normalize_phone]           # Exact after normalization
    - fields: [name]
      transformations: [soundex]                   # Phonetic — catches sound-alikes

Choosing a Strategy

Data Quality	Record Count	Recommended Strategy
High (clean emails/phones)	Any	composite with exact keys
Medium (some nulls, typos)	< 100K	composite with phonetic + exact keys
Medium	100K – 1M	composite with phonetic + domain + exact keys
Low (many nulls, inconsistent)	< 100K	composite with phonetic + sorted neighborhood
Low	1M+	lsh combined with exact keys

How to Choose Blocking Keys

1. Start with your strongest identifier. If 80% of records have email, block on email first.
2. Add a fallback for nulls. If 20% lack email, add a phone or name-based key.
3. Use transformations on fuzzy fields. Don't block on raw names — use soundex or normalize_name.
4. Measure recall. Run a sample, check how many known matches land in the same block.

Recall vs Performance

Blocking is a precision-recall trade-off:

• Tight blocking (exact email only): very fast, but misses records with typos, nulls, or different emails
• Loose blocking (phonetic name + first initial): high recall, but creates large blocks and more comparisons
• Balanced blocking (composite with 2-3 keys): the recommended default — exact identifiers for speed, fuzzy keys for coverage

   Recall
    100% ┤                            ● LSH (all fields)
         │                     ● Phonetic + Domain
     95% ┤              ● Composite (email + phone + soundex)
         │        ● Exact email + phone
     90% ┤  ● Exact email only
         │
     85% ┤
         └──────────────────────────────────────
              Fast                          Slow
                      Performance

The goal is the top-left corner: high recall with acceptable performance. Start with exact keys, measure recall, and add fuzzy keys until you reach your target.

Next Steps

• String Comparators — Algorithms that run after blocking produces candidate pairs
• Scoring Strategies — How comparison scores combine into match decisions
• Survivorship — Building golden records from matched pairs