Tutorial: Payment Deduplication

Detect duplicate payments across Stripe, Square, and an internal ledger. Payments that represent the same transaction often appear in multiple systems with slightly different amounts (processing fees), timestamps (timezone drift), and descriptions. This tutorial shows how to use range-based matching rules to catch them.

Time: 15 minutes Prerequisites: Python 3.9+, pip install kanoniv

What You Will Build

1. Create three payment data sources with intentional duplicates
2. Write matching rules using amount tolerance and date range windows
3. Combine rules into a composite matching strategy
4. Run reconciliation to identify duplicate payment clusters
5. Export duplicate groups as a pandas DataFrame for review

---

Step 1: Create Sample Data

Create three CSV files representing payment records from different processors. The same real-world payment may appear in multiple systems with slightly different values.

stripe_payments.csv

transaction_id,amount,date,merchant,customer_email,currency
SP-1001,99.99,2026-01-15,Acme Cloud Services,[email protected],USD
SP-1002,250.00,2026-01-15,Globex Consulting,[email protected],USD
SP-1003,49.95,2026-01-16,Initech Software,[email protected],USD
SP-1004,1250.00,2026-01-17,Hooli Premium,[email protected],USD
SP-1005,75.00,2026-01-18,Pied Piper Storage,[email protected],USD
SP-1006,320.00,2026-01-19,Umbrella Pharma,[email protected],USD
SP-1007,15.99,2026-01-19,Coffee Shop,[email protected],USD
SP-1008,500.00,2026-01-20,Cyberdyne Hosting,[email protected],USD
SP-1009,89.00,2026-01-21,Meta Cortex Inc,[email protected],USD
SP-1010,175.50,2026-01-22,Stark Industries,[email protected],USD

square_payments.csv

transaction_id,amount,date,description,customer_email,currency
SQ-2001,100.00,2026-01-15,ACME CLOUD SVC,[email protected],USD
SQ-2002,250.00,2026-01-16,GLOBEX CONSULT,[email protected],USD
SQ-2003,49.95,2026-01-16,INITECH SW,[email protected],USD
SQ-2004,1249.50,2026-01-17,HOOLI PREM,[email protected],USD
SQ-2005,75.25,2026-01-18,PIED PIPER STOR,[email protected],USD
SQ-2006,15.99,2026-01-20,COFFEE SHOP,[email protected],USD
SQ-2007,502.50,2026-01-20,CYBERDYNE HOST,[email protected],USD
SQ-2008,89.00,2026-01-21,META CORTEX,[email protected],USD
SQ-2009,42.00,2026-01-23,WAYNETECH,[email protected],USD
SQ-2010,175.50,2026-01-22,STARK IND,[email protected],USD

internal_ledger.csv

entry_id,amount,date,vendor,contact_email,currency
IL-3001,99.99,2026-01-15,Acme Cloud,[email protected],USD
IL-3002,250.00,2026-01-15,Globex,[email protected],USD
IL-3003,1250.00,2026-01-18,Hooli,[email protected],USD
IL-3004,74.75,2026-01-19,Pied Piper,[email protected],USD
IL-3005,320.00,2026-01-19,Umbrella,[email protected],USD
IL-3006,500.00,2026-01-20,Cyberdyne,[email protected],USD
IL-3007,89.50,2026-01-22,Meta Cortex,[email protected],USD
IL-3008,175.50,2026-01-22,Stark Industries,[email protected],USD

Key things to notice in this data:

Scenario	Stripe	Square	Ledger	Notes
Acme payment	$99.99 on Jan 15	$100.00 on Jan 15	$99.99 on Jan 15	Amount differs by $0.01
Globex payment	$250.00 on Jan 15	$250.00 on Jan 16	$250.00 on Jan 15	Date differs by 1 day
Hooli payment	$1250.00 on Jan 17	$1249.50 on Jan 17	$1250.00 on Jan 18	Amount and date differ
Pied Piper	$75.00 on Jan 18	$75.25 on Jan 18	$74.75 on Jan 19	Small amount spread
Coffee Shop	$15.99 on Jan 19	$15.99 on Jan 20	—	Same person, different date
Cyberdyne	$500.00 on Jan 20	$502.50 on Jan 20	$500.00 on Jan 20	Square has processing fee

---

Step 2: Write the Spec

Create a file called payment-dedup.yaml:

api_version: kanoniv/v1
identity_version: payment-dedup-v1

entity:
  name: payment
  description: Detect duplicate payments across processors using tolerance matching

sources:
  - name: stripe
    adapter: csv
    location: stripe_payments.csv
    primary_key: transaction_id
    attributes:
      amount: amount
      date: date
      merchant: merchant
      email: customer_email

  - name: square
    adapter: csv
    location: square_payments.csv
    primary_key: transaction_id
    attributes:
      amount: amount
      date: date
      merchant: description
      email: customer_email

  - name: ledger
    adapter: csv
    location: internal_ledger.csv
    primary_key: entry_id
    attributes:
      amount: amount
      date: date
      merchant: vendor
      email: contact_email

rules:
  - name: email_exact
    type: exact
    field: email
    weight: 1.0

  - name: amount_close
    type: range
    field: amount
    tolerance: 0.05       # Values <= 1.0 are treated as percentages (5%)
    weight: 0.8

  - name: date_close
    type: range
    field: date
    tolerance: 2           # Values > 1.0 are treated as absolute (2 days)
    weight: 0.7

survivorship:
  default: source_priority
  overrides:
    - field: amount
      strategy: source_priority
      priority: [ledger, stripe, square]
    - field: date
      strategy: source_priority
      priority: [ledger, stripe, square]
    - field: email
      strategy: source_priority
      priority: [ledger, stripe, square]

decision:
  thresholds:
    match: 0.90
    review: 0.65

Understanding the Rules

This spec uses three rules that work together:

email_exact (weight 1.0) — The customer email must match exactly. This is the strongest signal — if the emails differ, the payments are almost certainly for different customers.

amount_close (weight 0.8) — The payment amounts must be within 5% of each other. Tolerance values <= 1.0 are treated as percentages, so 0.05 means 5%. A $100.00 payment will match anything from $95.00 to $105.00.

date_close (weight 0.7) — The payment dates must be within 2 days of each other. Tolerance values > 1.0 are treated as absolute values, so 2 means 2 days. This accounts for timezone differences, settlement delays, and batch processing schedules.

For two payments to match, they need a combined weighted score of 0.90 or above. In practice this means email must match (contributing 1.0 of the maximum ~2.5 score) plus at least one of the tolerance rules must also fire.

Why Tolerance Matching?

Traditional exact matching would miss $99.99 vs $100.00 or Jan 15 vs Jan 16. Tolerance-based range rules close this gap by defining an acceptable window of variation.

---

Step 3: Validate and Plan

from kanoniv import Spec, validate, plan

spec = Spec.from_file("payment-dedup.yaml")
validate(spec).raise_on_error()
print("Spec is valid!")

execution = plan(spec)
print(f"\nPlan hash: {execution.hash}")
print(f"Sources: {execution.source_count}")
print(f"Rules: {execution.rule_count}")
print(f"Estimated comparisons: {execution.estimated_comparisons}")

Expected output:

Spec is valid!

Plan hash: b8e4d1f3
Sources: 3
Rules: 3
Estimated comparisons: 480

---

Step 4: Run Reconciliation

from kanoniv import Source, reconcile

sources = [
    Source.from_csv("stripe", "stripe_payments.csv"),
    Source.from_csv("square", "square_payments.csv"),
    Source.from_csv("ledger", "internal_ledger.csv"),
]

result = reconcile(sources, spec)

print(f"Records processed: {result.total_records}")
print(f"Clusters found: {len(result.clusters)}")
print(f"Golden records: {len(result.golden_records)}")
print(f"Merge rate: {result.merge_rate:.1%}")

Expected output:

Records processed: 28
Clusters found: 12
Golden records: 12
Merge rate: 57.1%

---

Step 5: Inspect Duplicate Clusters

The most valuable output is the set of clusters where len(members) > 1 — these are the detected duplicates.

print("=== Detected Duplicate Payments ===\n")

for cluster in result.clusters:
    if len(cluster.members) > 1:
        total = sum(float(m.fields.get("amount", 0)) for m in cluster.members)
        print(f"Cluster {cluster.id} ({len(cluster.members)} records, total: ${total:,.2f}):")
        for member in cluster.members:
            print(f"  [{member.source:8s}] {member.fields['id']:10s}  "
                  f"${float(member.fields['amount']):>10,.2f}  "
                  f"{member.fields['date']}  "
                  f"{member.fields.get('email', '')}")
        print()

Expected output:

=== Detected Duplicate Payments ===

Cluster c-001 (3 records, total: $299.98):
  [stripe  ] SP-1001     $     99.99  2026-01-15  [email protected]
  [square  ] SQ-2001     $    100.00  2026-01-15  [email protected]
  [ledger  ] IL-3001     $     99.99  2026-01-15  [email protected]

Cluster c-002 (3 records, total: $750.00):
  [stripe  ] SP-1002     $    250.00  2026-01-15  [email protected]
  [square  ] SQ-2002     $    250.00  2026-01-16  [email protected]
  [ledger  ] IL-3002     $    250.00  2026-01-15  [email protected]

Cluster c-003 (2 records, total: $99.90):
  [stripe  ] SP-1003     $     49.95  2026-01-16  [email protected]
  [square  ] SQ-2003     $     49.95  2026-01-16  [email protected]

Cluster c-004 (3 records, total: $3,749.50):
  [stripe  ] SP-1004     $  1,250.00  2026-01-17  [email protected]
  [square  ] SQ-2004     $  1,249.50  2026-01-17  [email protected]
  [ledger  ] IL-3003     $  1,250.00  2026-01-18  [email protected]

Cluster c-005 (3 records, total: $225.00):
  [stripe  ] SP-1005     $     75.00  2026-01-18  [email protected]
  [square  ] SQ-2005     $     75.25  2026-01-18  [email protected]
  [ledger  ] IL-3004     $     74.75  2026-01-19  [email protected]

Cluster c-006 (2 records, total: $31.98):
  [stripe  ] SP-1007     $     15.99  2026-01-19  [email protected]
  [square  ] SQ-2006     $     15.99  2026-01-20  [email protected]

Cluster c-007 (3 records, total: $1,502.50):
  [stripe  ] SP-1008     $    500.00  2026-01-20  [email protected]
  [square  ] SQ-2007     $    502.50  2026-01-20  [email protected]
  [ledger  ] IL-3006     $    500.00  2026-01-20  [email protected]

Cluster c-008 (3 records, total: $527.50):
  [stripe  ] SP-1010     $    175.50  2026-01-22  [email protected]
  [square  ] SQ-2010     $    175.50  2026-01-22  [email protected]
  [ledger  ] IL-3008     $    175.50  2026-01-22  [email protected]

---

Step 6: Export to DataFrame

Convert the results into a pandas DataFrame for further analysis or integration with your reporting tools.

df = result.to_pandas()

# Show only the duplicate groups (source_count > 1)
duplicates = df[df["source_count"] > 1]
print(f"Duplicate payment groups: {len(duplicates)}")
print(f"Total overpayment risk: ${duplicates['amount'].astype(float).sum():,.2f}")
print()
print(duplicates[["golden_id", "amount", "date", "email", "source_count"]].to_string(index=False))

Expected output:

Duplicate payment groups: 8
Total overpayment risk: $2,416.43

golden_id   amount        date                   email  source_count
    g-001    99.99  2026-01-15     [email protected]             3
    g-002   250.00  2026-01-15  [email protected]             3
    g-003    49.95  2026-01-16      [email protected]             2
    g-004  1250.00  2026-01-17        [email protected]             3
    g-005    75.00  2026-01-18     [email protected]             3
    g-006    15.99  2026-01-19     [email protected]             2
    g-007   500.00  2026-01-20  [email protected]            3
    g-008   175.50  2026-01-22       [email protected]             3

---

Step 7: Analyze Match Scores

You can inspect the score breakdown for each cluster to understand why records matched.

for cluster in result.clusters:
    if len(cluster.members) > 1:
        print(f"Cluster {cluster.id}:")
        for pair in cluster.pairs:
            print(f"  {pair.left_id} <-> {pair.right_id}")
            print(f"    Score: {pair.score:.3f}")
            for rule_name, rule_score in pair.rule_scores.items():
                status = "FIRED" if rule_score > 0 else "miss"
                print(f"    {rule_name}: {rule_score:.3f} ({status})")
        print()

Expected output (first cluster):

Cluster c-001:
  SP-1001 <-> SQ-2001
    Score: 0.926
    email_exact: 1.000 (FIRED)
    amount_close: 0.800 (FIRED)
    date_close: 0.700 (FIRED)
  SP-1001 <-> IL-3001
    Score: 1.000
    email_exact: 1.000 (FIRED)
    amount_close: 0.800 (FIRED)
    date_close: 0.700 (FIRED)

The Acme payment matched perfectly across all three rules for the Stripe-Ledger pair (identical amounts and dates), while the Stripe-Square pair still scored 0.926 because all three rules fired even with the $0.01 difference.

---

Step 8: Handle Review Candidates

Records that score between the review threshold (0.65) and the match threshold (0.90) land in the review queue. Check for any ambiguous matches:

if result.review_candidates:
    print("=== Records Requiring Review ===\n")
    for candidate in result.review_candidates:
        print(f"  {candidate.left_id} <-> {candidate.right_id}  "
              f"(score: {candidate.score:.3f})")
else:
    print("No records in the review queue.")

If the Meta Cortex payment ($89.00 in Stripe vs $89.50 in the ledger, with a 1-day date difference) falls below the match threshold but above the review threshold, it will appear here for manual review.

---

Full Working Script

from kanoniv import Source, Spec, reconcile, validate

# Load and validate
spec = Spec.from_file("payment-dedup.yaml")
validate(spec).raise_on_error()

# Load sources
sources = [
    Source.from_csv("stripe", "stripe_payments.csv"),
    Source.from_csv("square", "square_payments.csv"),
    Source.from_csv("ledger", "internal_ledger.csv"),
]

# Reconcile
result = reconcile(sources, spec)

# Summary
print(f"Records: {result.total_records}")
print(f"Clusters: {len(result.clusters)}")
print(f"Merge rate: {result.merge_rate:.1%}")

# Duplicate clusters
print("\n=== Duplicate Payments ===\n")
for cluster in result.clusters:
    if len(cluster.members) > 1:
        print(f"Cluster {cluster.id}:")
        for m in cluster.members:
            print(f"  [{m.source}] {m.fields['id']}  "
                  f"${float(m.fields['amount']):,.2f}  {m.fields['date']}")
        print()

# DataFrame export
df = result.to_pandas()
dupes = df[df["source_count"] > 1]
print(f"Duplicate groups: {len(dupes)}")
print(dupes.to_string(index=False))

---

Next Steps

• Customer 360 — Fuzzy name matching across CRM, billing, and support
• Lead Matching — Review queues and most-complete survivorship
• Spec Evolution — Version, diff, and validate specs in CI/CD
• Spec Reference — Full documentation on range rules and tolerance types