ML Training Data Versioning

Identity resolution specs are like models — they have parameters (weights, thresholds), structure (rules, blocking), and output (golden records). When the spec changes, the golden records change, and any ML model trained on those records may degrade silently. This page covers how to pin training data to spec versions and detect when spec changes require model retraining.

The Problem

A concrete scenario: your team trains a churn prediction model on golden records produced by Kanoniv. The model has been running at 0.89 precision for months. Then someone lowers match_threshold from 0.85 to 0.75 to capture more fuzzy matches. Two weeks later, precision drops to 0.72 and nobody can explain why.

What happened:

1. The lower threshold merged records that were previously separate entities
2. Entity boundaries shifted — some "customers" are now composites of two people
3. Feature distributions changed (average order count per customer went up, because records were combined)
4. The model was retrained on this new data without anyone noticing the spec had changed

The fix is simple: record which spec version produced which training data.

How Spec Changes Affect Training Data

Different types of spec changes have different downstream effects on golden records:

Spec Change	Effect on Golden Records	ML Impact
Lower match_threshold	More aggressive merging → fewer, larger entities	Feature distributions shift (aggregated metrics increase)
Raise match_threshold	More conservative merging → more, smaller entities	Entity splits may introduce duplicates in training data
Add a new source	New fields and records flow into golden records	New nulls in features, changed entity composition
Remove a source	Fields from that source disappear	Missing features, potential data gaps
Change survivorship strategy	Different source wins field conflicts	Field values change even though the same records are matched
Add/remove matching rules	Different records match or stop matching	Entity boundaries change fundamentally
Change blocking strategy	Different candidate pairs are evaluated	Previously unreachable matches may now occur (or vice versa)

Pinning Training Data to a Spec Version

The core pattern: every time you export golden records for ML training, record the plan_hash that produced them.

1. Compute and Record the Hash

from kanoniv import Spec, Source, plan, reconcile

spec = Spec.from_file("customer-v3.yaml")
p = plan(spec)

sources = [
    Source.from_csv("crm", "crm_contacts.csv"),
    Source.from_csv("billing", "stripe_customers.csv"),
]

result = reconcile(sources, spec)
df = result.to_pandas()

# Save with hash in filename
output_path = f"training_data_{p.plan_hash[:12]}.parquet"
df.to_parquet(output_path)

2. Save a Manifest

Store provenance metadata alongside the dataset:

import json
from datetime import datetime, timezone

manifest = {
    "plan_hash": p.plan_hash,
    "identity_version": spec.version,
    "created_at": datetime.now(timezone.utc).isoformat(),
    "entity_count": len(result.golden_records),
    "cluster_count": result.cluster_count,
    "merge_rate": result.merge_rate,
    "source_files": ["crm_contacts.csv", "stripe_customers.csv"],
    "output_file": output_path,
}

with open(f"manifest_{p.plan_hash[:12]}.json", "w") as f:
    json.dump(manifest, f, indent=2)

Now you can always answer: "which spec produced this training data?" by looking at the manifest.

Tracing Model Performance to Spec Changes

When model metrics degrade, use diff() to trace the regression:

from kanoniv import Spec, diff

# The spec that produced the good training data
old_spec = Spec.from_file("customer-v2.yaml")
# The current spec
new_spec = Spec.from_file("customer-v3.yaml")

changes = diff(old_spec, new_spec)

print(f"Changes detected: {changes.has_changes}")
print(f"Summary: {changes.summary}")

if changes.thresholds_changed:
    print("\nThreshold changes (likely affects entity boundaries):")
    for c in changes.decision_changes:
        print(f"  {c['path']}: {c['old_value']} -> {c['new_value']}")

if changes.survivorship_changed:
    print("\nSurvivorship changes (likely affects field values):")
    for c in changes.survivorship_changes:
        print(f"  {c['path']}: {c['old_value']} -> {c['new_value']}")

if changes.rules_added or changes.rules_removed:
    print("\nRule changes (likely affects entity boundaries):")
    if changes.rules_added:
        print(f"  Added: {changes.rules_added}")
    if changes.rules_removed:
        print(f"  Removed: {changes.rules_removed}")

This tells you exactly what changed and gives you a starting point for understanding the model degradation.

The Spec as a Model

The analogy is precise:

ML Model	Identity Spec
Model parameters (weights, biases)	Rule weights, thresholds, similarity cutoffs
Model architecture (layers, activations)	Rules, blocking strategy, survivorship
Model output (predictions)	Golden records (canonical entities)
Model checksum	plan_hash (SHA-256 of compiled plan)
Model comparison	diff(spec_a, spec_b)
Model versioning	identity_version + plan_hash

Treat the spec with the same discipline you'd treat a model checkpoint:

• Version it in git
• Record the hash with every output
• Diff before deploying changes
• Gate deployments on impact analysis

Full Workflow: Spec Change to Model Retrain

An end-to-end script that detects spec changes, compares reconciliation output, and saves the dataset with full provenance:

import json
from datetime import datetime, timezone
from kanoniv import Spec, Source, plan, diff, reconcile

# Load specs
old_spec = Spec.from_file("customer-v2.yaml")
new_spec = Spec.from_file("customer-v3.yaml")

# Check if the plan hash changed
old_plan = plan(old_spec)
new_plan = plan(new_spec)

if old_plan.plan_hash == new_plan.plan_hash:
    print("Plan hash unchanged — no retraining needed.")
    exit(0)

# Diff to understand what changed
changes = diff(old_spec, new_spec)
print(f"Spec changed: {changes.summary}")

# Reconcile with both specs to compare output
sources = [
    Source.from_csv("crm", "crm_contacts.csv"),
    Source.from_csv("billing", "stripe_customers.csv"),
]

old_result = reconcile(sources, old_spec)
new_result = reconcile(sources, new_spec)

# Compare key metrics
old_count = len(old_result.golden_records)
new_count = len(new_result.golden_records)
entity_change_pct = abs(new_count - old_count) / max(old_count, 1)

print(f"\nEntity count: {old_count} -> {new_count} ({entity_change_pct:+.1%})")
print(f"Merge rate: {old_result.merge_rate:.3f} -> {new_result.merge_rate:.3f}")

# Warn if entity count changed significantly
if entity_change_pct > 0.05:
    print("\nWARNING: Entity count changed by >5%. "
          "Review before retraining.")

# Save dataset with provenance
df = new_result.to_pandas()
output_path = f"training_data_{new_plan.plan_hash[:12]}.parquet"
df.to_parquet(output_path)

manifest = {
    "plan_hash": new_plan.plan_hash,
    "previous_plan_hash": old_plan.plan_hash,
    "identity_version": new_spec.version,
    "created_at": datetime.now(timezone.utc).isoformat(),
    "entity_count": new_count,
    "previous_entity_count": old_count,
    "merge_rate": new_result.merge_rate,
    "entity_change_pct": entity_change_pct,
    "diff_summary": changes.summary,
    "rules_added": changes.rules_added,
    "rules_removed": changes.rules_removed,
    "thresholds_changed": changes.thresholds_changed,
    "survivorship_changed": changes.survivorship_changed,
    "output_file": output_path,
}

with open(f"manifest_{new_plan.plan_hash[:12]}.json", "w") as f:
    json.dump(manifest, f, indent=2)

print(f"\nSaved: {output_path}")
print(f"Manifest: manifest_{new_plan.plan_hash[:12]}.json")

Integration with MLflow / W&B

Log spec provenance as experiment metadata alongside your model training runs:

MLflow

import mlflow

with mlflow.start_run():
    # Log spec provenance
    mlflow.log_param("plan_hash", new_plan.plan_hash)
    mlflow.log_param("identity_version", new_spec.version)
    mlflow.log_metric("entity_count", new_count)
    mlflow.log_metric("merge_rate", new_result.merge_rate)
    mlflow.log_metric("entity_change_pct", entity_change_pct)

    # Log the manifest as an artifact
    mlflow.log_artifact(f"manifest_{new_plan.plan_hash[:12]}.json")

    # ... train your model ...

Weights & Biases

import wandb

wandb.init(project="churn-model", config={
    "plan_hash": new_plan.plan_hash,
    "identity_version": new_spec.version,
    "entity_count": new_count,
    "merge_rate": new_result.merge_rate,
    "diff_summary": changes.summary,
})

# ... train your model ...

Now when model performance changes, you can filter experiment runs by plan_hash to see exactly which spec version produced the training data.

Guardrails: Detecting Risky Spec Changes

Add a CI/CD check that warns when a spec change is likely to affect training data significantly:

"""ci_spec_guard.py — Run in CI before deploying spec changes."""
import sys
from kanoniv import Spec, plan, diff

old_spec = Spec.from_file(sys.argv[1])  # production spec
new_spec = Spec.from_file(sys.argv[2])  # proposed spec

old_plan = plan(old_spec)
new_plan = plan(new_spec)

if old_plan.plan_hash == new_plan.plan_hash:
    print("PASS: No logical changes.")
    sys.exit(0)

changes = diff(old_spec, new_spec)
warnings = []

if changes.thresholds_changed:
    warnings.append(f"Thresholds changed — entity boundaries will shift")

if changes.survivorship_changed:
    warnings.append(f"Survivorship changed — golden record field values will change")

if changes.rules_added or changes.rules_removed:
    warnings.append(
        f"Rules changed ({len(changes.rules_added)} added, "
        f"{len(changes.rules_removed)} removed) — matching behavior will change")

if changes.sources_added or changes.sources_removed:
    warnings.append(
        f"Sources changed ({len(changes.sources_added)} added, "
        f"{len(changes.sources_removed)} removed) — data composition will change")

if warnings:
    print("WARN: Spec changes may affect ML training data:")
    for w in warnings:
        print(f"  - {w}")
    print(f"\nDiff summary: {changes.summary}")
    print("\nConsider retraining downstream models after this deploy.")
    sys.exit(0)  # warning, not failure

print("PASS: Changes are unlikely to affect training data.")

Usage in CI:

python ci_spec_guard.py specs/production.yaml specs/proposed.yaml

See Also

• Spec Versioning — How plan_hash and diff() work
• Spec Evolution Tutorial — CI/CD pipeline for spec validation
• Versioning Overview — The four versioning layers