How We Built a Rust-Based Identity Resolution Engine

Published February 2026 · 11 min read

When we started building Kanoniv, we had a choice: write the reconciliation engine in Python (fast to prototype, easy to hire for) or Rust (fast to execute, harder to build). We chose Rust. This post explains why, how the engine works, and what we learned.

Why Rust for Identity Resolution

Identity resolution is a CPU-bound problem. The core operation — comparing pairs of records field-by-field using string similarity algorithms — runs billions of times on moderately sized datasets. A 100K-record reconciliation with reasonable blocking generates millions of candidate pairs, each requiring multiple Jaro-Winkler or Levenshtein comparisons.

Python is too slow for this inner loop. Even with NumPy and Cython optimizations, pure Python string similarity runs 50-100x slower than native code. Libraries like Splink solve this by pushing computation to DuckDB or Spark — effectively escaping Python for the hot path.

We wanted the engine to be embeddable in a Python library (pip install, no external dependencies) while running at native speed. Rust + PyO3 gives us both.

The Performance Numbers

Benchmarks on a standard dataset (100K records, 3 sources, 5 matching rules):

Implementation	Time	Memory
Pure Python (thefuzz)	847s	2.1 GB
Python + NumPy vectorized	128s	3.4 GB
Splink (DuckDB)	14s	1.8 GB
Kanoniv (Rust via PyO3)	8s	420 MB

The Rust engine is ~100x faster than pure Python and ~60% faster than Splink's DuckDB backend on this workload. Memory usage is 5x lower than Python because Rust doesn't carry garbage collection overhead or Python object headers.

Architecture

The engine has four layers:

Python SDK (kanoniv)
  │  ← PyO3 FFI boundary
  ▼
Spec Parser
  │  ← YAML → typed IR
  ▼
Reconciliation Pipeline
  │  ← Normalize → Block → Compare → Cluster → Survive
  ▼
Output Builder
  │  ← Golden records, match pairs, diagnostics
  ▼
Python DataFrames

Layer 1: Spec Parser

The YAML spec is parsed into a typed intermediate representation (IR). Every field, rule, threshold, and strategy is validated at parse time — if the spec is invalid, you get a clear error before any data is processed.

#[derive(Deserialize, Validate)]
pub struct SpecIR {
    pub entity: EntityDef,
    pub sources: Vec<SourceDef>,
    pub rules: Vec<RuleDef>,
    pub survivorship: SurvivorshipDef,
    pub decision: DecisionDef,
}

The IR is owned by cannon-common and shared across the workspace. This means the API server, worker, and SDK all parse specs identically — no divergence between local and cloud execution.

Layer 2: Normalization

Each field value is normalized based on its type before comparison:

pub fn normalize(value: &str, field_type: FieldType) -> String {
    match field_type {
        FieldType::Name => normalize_name(value),
        FieldType::Email => normalize_email(value),
        FieldType::Phone => normalize_phone(value),
        FieldType::Address => normalize_address(value),
        FieldType::Generic => normalize_generic(value),
    }
}

fn normalize_name(value: &str) -> String {
    value
        .to_lowercase()
        .replace(&['.',  ',', '-'][..], " ")
        .split_whitespace()
        .filter(|w| !SUFFIXES.contains(w))  // Jr, Sr, III, etc.
        .collect::<Vec<_>>()
        .join(" ")
}

Normalization is deterministic — the same input always produces the same output. Original values are preserved for golden record creation.

Layer 3: Blocking

Blocking reduces the comparison space from O(n^2) to O(n * k) where k is the average block size.

We use multi-pass blocking: each rule defines its own blocking strategy. The union of all blocks forms the candidate set.

pub fn generate_blocks(records: &[Record], rules: &[RuleDef]) -> Vec<CandidatePair> {
    let mut candidates = HashSet::new();

    for rule in rules {
        let blocks = build_blocks(records, &rule.blocking_key);
        for block in blocks.values() {
            for (i, j) in block.pairs() {
                candidates.insert((i, j));
            }
        }
    }

    candidates.into_iter().collect()
}

Block keys are derived from the matching fields — an exact email rule blocks on email, a name + address rule blocks on (first 3 chars of last name + zip code).

Layer 4: Comparison

This is the hot loop — where most CPU time is spent. Each candidate pair is compared field-by-field using the appropriate similarity function.

Jaro-Winkler (used for names):

The Jaro-Winkler algorithm computes character-level similarity with a bonus for matching prefixes. Our implementation uses SIMD-friendly memory layout — record fields are stored in contiguous arrays rather than scattered across heap-allocated strings.

pub fn jaro_winkler(a: &str, b: &str) -> f64 {
    if a == b { return 1.0; }
    if a.is_empty() || b.is_empty() { return 0.0; }

    let jaro = jaro_similarity(a, b);

    // Winkler prefix bonus (up to 4 chars)
    let prefix_len = a.chars()
        .zip(b.chars())
        .take(4)
        .take_while(|(ca, cb)| ca == cb)
        .count();

    jaro + (prefix_len as f64 * 0.1 * (1.0 - jaro))
}

Parallelism: Comparisons within a block are independent, so we parallelize across blocks using Rayon. On an 8-core machine, this gives ~6x speedup for the comparison phase.

use rayon::prelude::*;

let results: Vec<MatchResult> = candidate_pairs
    .par_iter()
    .map(|(i, j)| compare_pair(&records[*i], &records[*j], rules))
    .filter(|r| r.score >= decision.review_threshold)
    .collect();

Layer 5: Clustering

Matched pairs are grouped into clusters using connected components. We use a union-find data structure for O(alpha(n)) amortized time per operation:

pub struct UnionFind {
    parent: Vec<usize>,
    rank: Vec<usize>,
}

impl UnionFind {
    pub fn union(&mut self, a: usize, b: usize) {
        let ra = self.find(a);
        let rb = self.find(b);
        if ra == rb { return; }
        // Union by rank
        match self.rank[ra].cmp(&self.rank[rb]) {
            Ordering::Less => self.parent[ra] = rb,
            Ordering::Greater => self.parent[rb] = ra,
            Ordering::Equal => {
                self.parent[rb] = ra;
                self.rank[ra] += 1;
            }
        }
    }
}

Layer 6: Survivorship

The survivorship engine selects the best field values for each golden record based on the configured strategy:

pub fn apply_survivorship(
    cluster: &[Record],
    strategy: &SurvivorshipDef,
) -> GoldenRecord {
    match strategy.strategy {
        Strategy::SourcePriority => {
            let priority = &strategy.priority;
            let mut golden = GoldenRecord::new();
            // For each field, take the first non-null value
            // from the highest-priority source
            for source_name in priority {
                for record in cluster.iter().filter(|r| r.source == *source_name) {
                    golden.fill_nulls_from(record);
                }
            }
            golden
        }
        Strategy::MostRecent => { /* ... */ }
        Strategy::MostComplete => { /* ... */ }
    }
}

PyO3: Bridging Rust and Python

PyO3 lets us expose Rust functions as native Python modules. The Python SDK calls into Rust for all heavy computation:

#[pyfunction]
fn reconcile(sources: Vec<PySource>, spec: PySpec) -> PyResult<PyReconcileResult> {
    // Convert Python types to Rust types
    let rust_sources = sources.iter().map(|s| s.to_rust()).collect();
    let rust_spec = spec.to_rust()?;

    // Run the Rust pipeline
    let result = engine::reconcile(&rust_sources, &rust_spec)
        .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, _>(e.to_string()))?;

    // Convert back to Python types
    Ok(PyReconcileResult::from_rust(result))
}

#[pymodule]
fn _kanoniv_core(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(reconcile, m)?)?;
    m.add_function(wrap_pyfunction!(validate, m)?)?;
    Ok(())
}

The data crossing boundary is the main cost. We minimize it by:

1. Accepting CSV paths (Rust reads the file directly) rather than passing DataFrames row-by-row
2. Returning results as columnar data that PyArrow/Pandas can consume without copying
3. Keeping all intermediate data in Rust — only the final results cross the FFI boundary

Building Wheels with Maturin

Maturin handles the complexity of building Python wheels from Rust code. Our CI pipeline builds wheels for three platforms:

# .github/workflows/release-sdk.yml
strategy:
  matrix:
    include:
      - os: ubuntu-latest
        target: x86_64-unknown-linux-gnu
      - os: macos-13
        target: x86_64-apple-darwin
      - os: macos-14
        target: aarch64-apple-darwin

steps:
  - uses: PyO3/maturin-action@v1
    with:
      target: ${{ matrix.target }}
      args: --release --out dist

Users get a pre-built native binary — pip install kanoniv just works. No Rust toolchain needed.

Lessons Learned

1. Rust's Ownership Model Fits Record Processing

Records flow through the pipeline in a single direction: normalize → block → compare → cluster → survive. Each stage takes ownership of records, transforms them, and passes them to the next stage. This maps cleanly to Rust's ownership model — no shared mutable state, no data races.

2. String Handling Is the Bottleneck

Identity resolution is fundamentally a string comparison problem. We spent more time optimizing string operations than any other part of the engine:

• Pre-computing character arrays for Jaro-Winkler instead of iterating over UTF-8 bytes
• Using byte-level comparisons for ASCII-only fields (email, phone)
• Caching normalized forms to avoid redundant normalization

3. PyO3 Ergonomics Have Improved Dramatically

Early PyO3 was painful — lots of manual type conversion and error handling. Modern PyO3 (0.20+) with #[pyclass], #[pymethods], and automatic type conversion is pleasant to work with. The FFI boundary is almost invisible in the Python SDK code.

4. Testing Across the FFI Boundary Is Essential

We maintain three test suites:

• Rust unit tests: Test the engine internals (similarity functions, blocking, clustering)
• Python integration tests: Test the PyO3 bindings (round-trip data, error handling)
• Contract tests: Verify that the same spec produces the same results in Rust and in the cloud API

The contract tests caught several subtle bugs where Rust and Python handled edge cases differently (empty strings, Unicode normalization, floating-point comparison thresholds).

5. Memory Efficiency Matters More Than You'd Think

Identity resolution datasets are large, and the engine needs to hold all records in memory during comparison. Rust's zero-cost abstractions let us represent a record as a compact struct (a few hundred bytes) rather than a Python dict (thousands of bytes with object headers, hash tables, and reference counts). This 5-10x memory reduction means the engine can process larger datasets on the same hardware.

Why Not Just Use DuckDB?

Splink's approach — express matching as SQL and push it to DuckDB — is smart and works well. But there are trade-offs:

	Rust Engine	DuckDB/SQL
Custom similarity functions	Easy to add	Requires UDFs or extensions
Survivorship logic	Native	Requires complex SQL
Embedding in a library	Yes (single binary)	Requires DuckDB runtime
Memory control	Full control	DuckDB manages its own memory
Debugging	Standard Rust tooling	SQL explain plans

For Kanoniv, the critical factor was survivorship. Expressing survivorship as SQL is painful — it requires window functions, conditional aggregation, and custom tiebreaking logic for every field. In Rust, it's a straightforward function that iterates through a cluster and picks values.

What's Next

We're working on:

• Incremental reconciliation: Process only new/changed records instead of the full dataset
• SIMD-accelerated comparisons: Using AVX2/NEON for character-level operations
• WebAssembly target: Run the engine in the browser for instant spec validation

The Rust engine is the foundation that makes all of this possible. A Python engine would require rewriting the core for each new deployment target. The Rust engine compiles to native code, WebAssembly, or a Python extension from the same source.