Active Learning (Feedback Loop)

Kanoniv supports an active learning workflow where you label uncertain record pairs and feed those labels back into the reconciliation engine. The engine uses your feedback to refine its scoring weights, improving match quality with each iteration.

This is the fastest way to improve match quality when you have domain expertise but no pre-existing labeled dataset.

How It Works

The active learning loop has four steps:

1. Reconcile      ->  2. Surface uncertain pairs  ->  3. Label pairs
       ^                                                      |
       |                                                      v
       +------------------  4. Re-reconcile with feedback  ---+

1. Reconcile - Run identity resolution with your current spec
2. Surface uncertain pairs - The engine identifies pairs closest to the decision boundary (the "review zone")
3. Label pairs - A human reviews the uncertain pairs and labels each as "match" or "no_match"
4. Re-reconcile - Run again with the labels. The engine adjusts scoring weights based on your feedback

Each cycle typically takes 10-20 minutes of labeling and produces measurable improvements in precision and recall.

Quick Start

from kanoniv import reconcile, FeedbackLabel, Source, Spec

spec = Spec.from_file("customer-spec.yaml")
sources = [
    Source.from_csv("crm", "crm_contacts.csv"),
    Source.from_csv("billing", "stripe_customers.csv"),
]

# Step 1: Initial reconciliation
result = reconcile(sources, spec)
print(f"Clusters: {result.cluster_count}, Merge rate: {result.merge_rate:.1%}")

# Step 2: Get uncertain pairs
pairs = result.uncertain_pairs(n=30)
for p in pairs[:5]:
    print(f"  {p['a']['name']} vs {p['b']['name']} - confidence: {p['confidence']:.2f}")

# Step 3: Label them (in practice, use a review UI or spreadsheet)
labels = [
    FeedbackLabel("cust_123", "cust_456", "crm", "billing", "match"),
    FeedbackLabel("cust_789", "cust_012", "crm", "billing", "no_match"),
    FeedbackLabel("cust_345", "cust_678", "crm", "billing", "match"),
]

# Step 4: Re-reconcile with feedback
refined = reconcile(sources, spec, feedback=labels)
print(f"Clusters: {refined.cluster_count}, Merge rate: {refined.merge_rate:.1%}")

# Evaluate improvement
refined.evaluate()

Uncertain Pairs

result.uncertain_pairs(n) returns the n pairs closest to the decision boundary. These are pairs that the engine is least confident about - the ones where human judgment adds the most value.

pairs = result.uncertain_pairs(n=30)

Each pair is a dictionary with the following structure:

{
    "a": {
        "id": "cust_123",
        "source": "crm",
        "name": "John Smith",
        "email": "[email protected]",
        "phone": "555-0100"
    },
    "b": {
        "id": "cust_456",
        "source": "billing",
        "name": "Jon Smith",
        "email": "[email protected]",
        "phone": None
    },
    "confidence": 0.73,
    "decision": "review",
    "rule_scores": {
        "email_exact": 1.0,
        "name_fuzzy": 0.92,
        "phone_exact": 0.0
    }
}

How pairs are selected

The engine ranks all pairs by their distance from the decision boundary:

• Pairs with confidence closest to the match threshold are selected first
• Then pairs closest to the review threshold
• Pairs that are clearly matches (0.99) or clearly non-matches (0.05) are excluded - labeling them adds no information

This is called uncertainty sampling and is the most efficient active learning strategy for record linkage.

FeedbackLabel

FeedbackLabel represents a human judgment about whether two records are the same entity.

from kanoniv import FeedbackLabel

label = FeedbackLabel(
    entity_a_id="cust_123",      # Record ID from source A
    entity_b_id="cust_456",      # Record ID from source B
    source_a="crm",              # Source system name for record A
    source_b="billing",          # Source system name for record B
    label="match"                # "match" or "no_match"
)

Parameters

Parameter	Type	Description
entity_a_id	str	Record ID from the first source
entity_b_id	str	Record ID from the second source
source_a	str	Name of the first source system (must match a source in your spec)
source_b	str	Name of the second source system (must match a source in your spec)
label	str	Human judgment: "match" or "no_match"

Label semantics

• "match" - These two records represent the same real-world entity. The engine should merge them.
• "no_match" - These two records represent different entities. The engine should keep them separate.

Behavior by Scoring Strategy

Active learning behaves differently depending on your spec's scoring strategy.

Fellegi-Sunter (probabilistic)

When using Fellegi-Sunter scoring, feedback labels are used to refine the m/u probability estimates through supervised EM training.

The engine blends labeled pair statistics with the existing m/u weights:

• Match labels increase the m-probability (P(agree | true match)) for fields that agreed in the labeled pair
• No-match labels increase the u-probability (P(agree | not a match)) for fields that agreed despite being different entities

This is the most powerful use of feedback - it directly calibrates the statistical model.

decision:
  scoring:
    strategy: fellegi_sunter
    training:
      method: em
      max_iterations: 25
    fields:
      - name: email
        comparator: exact
        m_probability: 0.95
        u_probability: 0.01
      - name: name
        comparator: jaro_winkler
        m_probability: 0.90
        u_probability: 0.10
  thresholds:
    match: 0.90
    review: 0.60

After a few rounds of feedback, the m/u estimates become much more accurate than the initial guesses, and the decision boundary shifts to the right place automatically.

Rules-based / Weighted Sum

When using rules-based scoring or weighted sum, feedback labels are treated as force-merge and force-split overrides:

• "match" labels act as force-merge overrides - the labeled pair is merged regardless of score
• "no_match" labels act as force-split overrides - the labeled pair is kept separate regardless of score

The engine does not adjust rule weights based on feedback. The labels function as persistent corrections, similar to overrides.

decision:
  scoring: weighted_sum
  thresholds:
    match: 0.85
    review: 0.60

This is still useful - you get the benefit of targeted labeling via uncertain_pairs(), and the corrections persist across runs - but it does not improve the scoring model itself.

Persisting Feedback

Feedback labels persist across reconciliation runs through save() and load():

# Run 1: Initial reconciliation + first round of labeling
result = reconcile(sources, spec)
pairs = result.uncertain_pairs(n=30)

# ... label pairs ...

result_with_feedback = reconcile(sources, spec, feedback=round_1_labels)
result_with_feedback.save("customer-reconciliation.json")

# Run 2: Load previous result, add more labels
previous = ReconcileResult.load("customer-reconciliation.json")
new_pairs = previous.uncertain_pairs(n=30)

# ... label new pairs ...

# Combine old + new labels
all_labels = round_1_labels + round_2_labels
refined = reconcile(sources, spec, feedback=all_labels)
refined.save("customer-reconciliation.json")

Accumulating labels across runs

Each round of labeling adds to your labeled dataset. The engine uses all accumulated labels when retraining:

from kanoniv import reconcile, FeedbackLabel, ReconcileResult, Source, Spec

spec = Spec.from_file("customer-spec.yaml")
sources = [
    Source.from_csv("crm", "crm.csv"),
    Source.from_csv("billing", "billing.csv"),
]

# Maintain a running list of all labels
all_labels = []

for round_num in range(1, 6):  # 5 rounds of active learning
    result = reconcile(sources, spec, feedback=all_labels)
    eval_result = result.evaluate()
    print(f"Round {round_num}: F1={eval_result.f1:.3f} "
          f"P={eval_result.precision:.3f} R={eval_result.recall:.3f}")

    # Get uncertain pairs for this round
    pairs = result.uncertain_pairs(n=20)
    if not pairs:
        print("No more uncertain pairs - model is confident")
        break

    # Label them (replace with your labeling workflow)
    new_labels = label_pairs(pairs)  # your labeling function
    all_labels.extend(new_labels)
    print(f"  Total labels: {len(all_labels)}")

A typical progression:

Round	Labels	Precision	Recall	F1
1 (no feedback)	0	0.94	0.87	0.90
2	20	0.96	0.89	0.92
3	40	0.97	0.91	0.94
4	60	0.97	0.93	0.95
5	80	0.98	0.94	0.96

Diminishing returns typically set in after 50-100 labels. If F1 plateaus, the remaining errors are likely caused by data quality issues or missing rules rather than threshold calibration.

Labeling Best Practices

1. Label the uncertain pairs, not random ones

uncertain_pairs() selects the pairs where your labels have the most impact. Labeling random pairs is roughly 5x less efficient at improving the model.

2. Label both matches and non-matches

A balanced label set (roughly 50/50 match/no_match) produces the best weight calibration. If you only label matches, the engine cannot learn what non-matches look like.

3. Start with 20-30 labels per round

This is enough to shift the decision boundary without overwhelming the labeler. You can always do more rounds.

4. Use the rule scores to speed up labeling

The rule_scores in each uncertain pair show which fields agreed and which did not. This helps you make faster judgments:

for p in pairs:
    print(f"\n{p['a']['name']} vs {p['b']['name']}")
    print(f"  Confidence: {p['confidence']:.2f}")
    for rule, score in p['rule_scores'].items():
        indicator = "Y" if score > 0.8 else "N" if score < 0.2 else "?"
        print(f"  [{indicator}] {rule}: {score:.2f}")

John Smith vs Jon Smith
  Confidence: 0.73
  [Y] email_exact: 1.00
  [Y] name_fuzzy: 0.92
  [N] phone_exact: 0.00

Email and name agree, phone disagrees. If you know this person recently changed phone numbers, it is a match.

5. Resolve disagreements with a second labeler

If two labelers disagree on a pair, discuss it. Disagreements often reveal ambiguity in your matching criteria that should be resolved at the spec level (e.g., "do we match people who share a company email alias?").

Combining with Graph Clustering

Active learning and graph-based clustering complement each other well:

• Graph clustering prevents weak transitive chains at the cluster level
• Active learning refines the scoring model at the pair level

Use both together for the best results:

decision:
  scoring:
    strategy: fellegi_sunter
    training:
      method: em
      max_iterations: 25
    fields:
      - name: email
        comparator: exact
        m_probability: 0.95
        u_probability: 0.01
      - name: name
        comparator: jaro_winkler
        m_probability: 0.90
        u_probability: 0.10
      - name: phone
        comparator: exact
        m_probability: 0.90
        u_probability: 0.02
  thresholds:
    match: 0.85
    review: 0.60
  clustering:
    method: graph
    min_edge_weight: 0.70
    min_cluster_coherence: 0.60

Then run the active learning loop on top:

result = reconcile(sources, spec, feedback=labels)

The engine applies feedback to refine FS weights, then uses graph clustering on the refined scores. The two features operate on different stages of the pipeline and do not interfere with each other.

Cloud Workflow

Cloud feedback is store-now, apply-later

Feedback labels uploaded to the cloud via client.feedback.create() are persisted for record-keeping but are not yet automatically applied during cloud reconciliation. The cloud engine does not use feedback labels to adjust scoring weights. This is planned for a future release.

The recommended pattern is a hybrid workflow: run the active learning loop locally for fast iteration, then use the cloud for production reconciliation and feedback storage.

Hybrid Active Learning Pattern

import kanoniv
from kanoniv import reconcile, FeedbackLabel, Source, Spec, Client

spec = Spec.from_file("customer-spec.yaml")
sources = [
    Source.from_csv("crm", "crm_contacts.csv"),
    Source.from_csv("billing", "stripe_customers.csv"),
]

# ── Phase 1: Local active learning loop ──────────────────────
all_labels = []

for round_num in range(1, 4):
    result = reconcile(sources, spec, feedback=all_labels)
    print(f"Round {round_num}: clusters={result.cluster_count}, "
          f"merge_rate={result.merge_rate:.1%}")

    pairs = result.uncertain_pairs(n=20)
    if not pairs:
        print("No more uncertain pairs")
        break

    # Label pairs (replace with your review UI)
    new_labels = []
    for p in pairs:
        # ... human review ...
        new_labels.append(FeedbackLabel(
            p["a"]["id"], p["b"]["id"],
            p["a"]["source"], p["b"]["source"],
            "match",  # or "no_match"
        ))
    all_labels.extend(new_labels)

# Final local reconciliation with all feedback
final = reconcile(sources, spec, feedback=all_labels)
print(f"Final: clusters={final.cluster_count}, merge_rate={final.merge_rate:.1%}")


# ── Phase 2: Upload feedback to cloud for record-keeping ─────
with Client(api_key="kn_...") as client:
    client.feedback.create(labels=[
        {
            "entity_a_id": l.entity_a_id,
            "entity_b_id": l.entity_b_id,
            "source_a": l.source_a,
            "source_b": l.source_b,
            "label": l.label,
        }
        for l in all_labels
    ])
    print(f"Uploaded {len(all_labels)} feedback labels to cloud")


# ── Phase 3: Run production reconciliation on the cloud ──────
with kanoniv.cloud.reconcile(sources, spec, api_key="kn_...") as result:
    print(result.summary())

Feedback API

The cloud SDK provides client.feedback for managing feedback labels server-side:

from kanoniv import Client

with Client(api_key="kn_...") as client:
    # List existing labels
    labels = client.feedback.list(limit=100)
    print(f"{len(labels)} labels stored")

    # Create new labels
    client.feedback.create(labels=[
        {
            "entity_a_id": "cust_123",
            "entity_b_id": "cust_456",
            "source_a": "crm",
            "source_b": "billing",
            "label": "match",
            "reason": "Confirmed same person",
        },
    ])

    # Delete a label
    client.feedback.delete("label-uuid")

See the Cloud SDK reference for the full feedback API.

Next Steps

• Tuning Match Quality - Manual threshold tuning workflow
• Graph-based Clustering - Prevent weak transitive chains
• Scoring Strategies - Fellegi-Sunter, weighted sum, and ML ensemble
• Overrides - Manual force-merge and force-split corrections
• Arrow Data Plane - 6x faster warehouse-to-cloud ingest
• Cloud SDK - Full cloud API reference including feedback endpoints