Governance

Governance settings control metadata, data quality policies, exclusions, relations, and reference identifiers. These features range from simple informational metadata (available to all tiers) to enterprise-grade governance policies that enforce data quality standards.

Metadata

metadata:
  name: "Customer Identity Resolution"
  description: "Production customer identity resolution spec"
  owner: [email protected]
  tags: [production, customer, v2]

Field	Type	Required	Default	Description
name	string	No	N/A	Human-readable name for this spec
description	string	No	N/A	Human-readable description of this spec's purpose
owner	string	No	N/A	Email or team identifier for spec ownership
tags	array[string]	No	[]	Arbitrary labels for filtering and organization

Metadata is informational and does not affect reconciliation behavior. It is useful for:

• Tracking spec ownership: know who to contact when a spec needs changes
• Filtering specs by tag: query specs by environment (production, staging), domain (customer, vendor), or version (v2, beta)
• Naming and describing specs: the name and description fields provide human-readable context

from kanoniv import Spec

spec = Spec.from_file("customer-spec.yaml")
metadata = spec.raw.get("metadata", {})
print(metadata.get("owner"))  # "[email protected]"
print(metadata.get("tags"))   # ["production", "customer", "v2"]

Tag Conventions

While tags are free-form, we recommend a consistent naming convention:

Category	Examples	Purpose
Environment	production, staging, development	Deployment target
Domain	customer, vendor, product, employee	Business domain
Version	v1, v2, beta, deprecated	Spec lifecycle stage
Team	data-eng, analytics, compliance	Owning team

Governance Policies (Cloud)

Governance policies enforce data quality standards across your specs. When enabled, these policies are checked during validation and reconciliation.

governance:
  require_freshness: true
  require_schema: true
  required_tags:
    - production

Field	Type	Required	Default	Description
require_freshness	boolean	No	false	All sources must have freshness configured
require_schema	boolean	No	false	All sources must have schema validation
required_tags	array[string]	No	[]	Every source must have all of these tags

Freshness Enforcement

When require_freshness is true, freshness is enforced at two stages:

1. Validation time. Every source in the spec must include a freshness block. If any source is missing it, the spec fails validation at upload time:

governance:
  require_freshness: true

sources:
  crm:
    adapter: csv
    location: data/contacts.csv
    primary_key: id
    freshness:
      max_age: "24h"           # Required when governance.require_freshness is true

  billing:
    adapter: csv
    location: data/invoices.csv
    primary_key: invoice_id
    # Missing freshness block -- validation will fail

from kanoniv import Spec, validate

spec = Spec.from_file("customer-spec.yaml")
result = validate(spec)
# ValidationError: Source 'billing' must have freshness configured
#   (governance.require_freshness is true)

2. Reconciliation time. When a job runs, the worker checks actual data freshness against each source's max_age before matching begins:

• Queries the most recent data timestamp for each source from the database
• Computes the age of each source and compares it to max_age_seconds
• Sources exceeding their max_age are flagged as stale
• If require_freshness is true and any source is stale, the pipeline aborts with a governance violation and the job is marked as failed
• If require_freshness is false (or not set), stale sources produce a warning in the health report but reconciliation continues

This two-layer enforcement ensures that (a) every source declares a freshness expectation, and (b) the actual data meets that expectation at run time. Without governance, freshness checks are advisory only — stale sources are logged but never block the pipeline.

Schema Enforcement

The require_schema policy has two effects:

1. At validation time: every source in the spec must include a schema block. Sources without a schema cause a validation error.
2. At reconciliation time: if field-presence drift is detected (missing or extra fields compared to the schema declaration), the pipeline aborts with a governance violation error instead of continuing with a warning.

governance:
  require_schema: true

sources:
  crm:
    adapter: csv
    location: data/contacts.csv
    primary_key: id
    schema:
      email: { type: string, pii: true }
      phone: { type: string, pii: true }
      name: { type: string }
      created_at: { type: string }

Drift detection works by sampling the first entity from each source and comparing its field names against the schema declaration. It checks for missing fields (declared in schema but absent in data) and extra fields (present in data but not declared in schema). This is a field-presence check, not a per-record type or nullability validation. No records are excluded based on schema checks.

Without require_schema, drift still produces a warning log but reconciliation continues normally. With require_schema: true, any detected drift causes the batch run to fail immediately.

Required Tags

When required_tags is configured, every source must include all of the listed tags. This enforces consistent tagging across data sources:

governance:
  required_tags:
    - production
    - reviewed

sources:
  crm:
    adapter: csv
    location: data/contacts.csv
    primary_key: id
    tags: [production, reviewed]     # Passes: has both required tags

  billing:
    adapter: csv
    location: data/invoices.csv
    primary_key: invoice_id
    tags: [production]               # Fails: missing 'reviewed' tag

Exclusions (Cloud)

Exclude records or record pairs from reconciliation. Exclusions are a tagged enum with a type field that determines the variant. There are three exclusion types: field_value, explicit_pair, and cross_tenant.

Exclusions are fully enforced at runtime in two places:

1. Pre-filter. Before blocking, entities matching any field_value exclusion are skipped entirely and never enter candidate pair generation.
2. Pair-level. During pair evaluation, all three exclusion types are checked. If either entity matches a field_value exclusion, or the pair matches an explicit_pair or cross_tenant exclusion, the pair is immediately rejected with a NoMerge decision.

Field Value Exclusions

Exclude records where a field matches a condition.

exclusions:
  - type: field_value
    field: status
    operator: "eq"
    value: "deleted"

  - type: field_value
    field: name
    operator: "contains"
    value: "test"

  - type: field_value
    field: phone
    operator: "is_empty"

Field	Type	Required	Description
type	string	Yes	Must be field_value
field	string	Yes	Field to check for exclusion
operator	string	Yes	Comparison operator (see table below)
value	any	Depends	Value to compare against (not required for is_empty)

Supported operators:

Operator	Description	Example
eq or ==	Exact string match	operator: "eq", value: "deleted"
ne or !=	Not equal	operator: "ne", value: "active"
contains	Field value contains substring	operator: "contains", value: "test"
is_empty	Field is missing or empty string	operator: "is_empty" (no value needed)

WARNING

Only the four operators above are implemented. Other operators (such as in, ends_with, starts_with) are not supported and will silently match nothing if used.

Explicit Pair Exclusions

Prevent two specific records from being matched, identified by their external_id values. Use this for known non-matches that you want to permanently exclude.

exclusions:
  - type: explicit_pair
    left_id_field: "customer-123"
    right_id_field: "customer-456"

  - type: explicit_pair
    left_id_field: "acme-corp-old"
    right_id_field: "acme-corp-new"

Field	Type	Required	Description
type	string	Yes	Must be explicit_pair
left_id_field	string	Yes	The external_id of the first record
right_id_field	string	Yes	The external_id of the second record

The engine checks both directions — if record A has external_id = "customer-123" and record B has external_id = "customer-456", the pair is excluded regardless of which is "left" and which is "right".

TIP

Despite the field names (left_id_field, right_id_field), these are literal external_id values, not field name lookups. Each entry excludes exactly one pair of records.

Cross-Tenant Exclusions

Prevent records from different tenants from being matched together. Useful in multi-tenant deployments where data isolation is required.

exclusions:
  - type: cross_tenant
    enabled: true

Field	Type	Required	Default	Description
type	string	Yes	N/A	Must be cross_tenant
enabled	boolean	No	false	Whether to prevent cross-tenant matching

When enabled, the engine compares the tenant_id of both records in a pair. If they differ, the pair is excluded.

Relations (Cloud)

Define relationships between different entity types. Relations enable match propagation across entity boundaries — when records of one type are matched, the engine can automatically propagate that decision to related records of another type.

relations:
  - name: company_contacts
    from_entity: customer
    to_entity: company
    join_key: company_id
    cardinality: many_to_one
    propagate_match: false

  - name: household_members
    from_entity: customer
    to_entity: household
    join_key: household_id
    cardinality: many_to_one
    propagate_match: true

  - name: account_invoices
    from_entity: invoice
    to_entity: customer
    join_key: customer_id
    cardinality: many_to_one

Field	Type	Required	Default	Description
name	string	Yes	N/A	Unique relation identifier
from_entity	string	Yes	N/A	Source entity type
to_entity	string	Yes	N/A	Target entity type
join_key	string	Yes	N/A	Field on from_entity records that references to_entity records
cardinality	string	Yes	N/A	one_to_one, one_to_many, many_to_one, many_to_many
propagate_match	boolean	No	false	Whether to propagate match decisions across this relationship

Cardinality

Cardinality describes the nature of the relationship between entity types. It is stored in the compiled plan for documentation and downstream tooling, but the engine does not enforce cardinality constraints at runtime — propagation works the same way regardless of the declared cardinality.

Type	Description	Example
one_to_one	Each record maps to exactly one record in the target	Employee to badge number
one_to_many	One record maps to multiple target records	Customer to orders
many_to_one	Multiple records map to one target record	Contacts to company
many_to_many	Multiple records map to multiple target records	Students to courses

Match Propagation

When propagate_match is true, the engine propagates match decisions across the relationship after the initial matching phase completes. Relations with propagate_match: false are stored in the plan but have no runtime effect.

How it works:

1. Scan matched clusters. After matching, the engine iterates each cluster of merged entities.

2. Collect join key values. For each cluster, the engine finds all from_entity records and collects their join_key field values. For example, if two customer records were matched and both have a company_id, those company IDs are collected.

3. Find target entities. The engine looks up to_entity records whose external_id or data values match the collected join key values.

4. Create merge decisions. If multiple target entities are found (e.g., two different company records linked by matched customers), the engine creates Merge decisions between them with confidence 1.0. Each decision is tagged with RELATION:{name}:hop:{n} for traceability.

5. Multi-hop propagation. The engine re-clusters after each hop and repeats up to 5 hops. This means if a relation propagation creates new matches, those matches can trigger further propagation through other relations. The engine stops early if a hop produces no new decisions.

6. Override-aware. Propagated decisions respect manual overrides. If a pair has been explicitly split or locked by an operator, the propagation will not override that decision.

Example: Two customer records (customer-A and customer-B) are matched by the rules engine. Both have household_id fields pointing to different household records (household-X and household-Y). With propagate_match: true on the household relation, the engine will also merge household-X and household-Y.

Linked Entities API

After reconciliation, the canonical entity detail endpoint returns all linked entities. When you query a canonical entity by ID, the response includes linked_entities — the external records that were merged into that canonical identity:

GET /v1/canonical/{id}

{
  "canonical": { "id": "...", "canonical_data": { ... } },
  "linked_entities": [
    { "id": "...", "source": "crm", "external_id": "customer-A", ... },
    { "id": "...", "source": "billing", "external_id": "customer-B", ... }
  ],
  "links": [ ... ]
}

Reference Identifiers (Cloud)

Map external identifier systems to your entity records. Reference identifiers enable deterministic matching: if two records share the same non-empty value for a reference identifier field, they are immediately merged with confidence 1.0. This is a short-circuit: when a reference ID matches, all other matching rules are skipped entirely.

reference_identifiers:
  - name: duns_number
    type: external
    authority: dnb
    field: duns
    match_weight: 1.0
    conditions:
      - field: duns
        operator: "not_empty"

  - name: lei_code
    type: external
    authority: gleif
    field: lei
    match_weight: 1.0

  - name: internal_crm_id
    type: internal
    field: crm_id
    match_weight: 0.8

Field	Type	Required	Default	Description
name	string	Yes	N/A	Identifier name (unique within the spec)
type	string	Yes	N/A	Identifier type: external or internal
authority	string	No	N/A	External system or registry (e.g., dnb, gleif, irs)
field	string	Yes	N/A	Field in your data containing this identifier
match_weight	number	No	1.0	Accepted for forward compatibility but does not currently affect scoring (see note below)
conditions	array[object]	No	[]	Accepted for forward compatibility but not evaluated at runtime (see note below)

How Reference ID Matching Works

Reference identifiers are evaluated before any matching rules run. The engine checks each reference identifier in order:

1. For each reference identifier, it looks up the identifier field value in both records
2. If both records have a non-empty value and the values are equal, the pair is immediately merged with confidence 1.0
3. No further rules are evaluated; this is a deterministic short-circuit

This means reference identifiers act as a fast path for high-confidence identity resolution. If you know that two records share the same DUNS number, there is no need to run fuzzy name matching or other heuristic rules.

match_weight and conditions are currently unused

The match_weight and conditions fields are accepted in the spec and stored in the plan, but neither affects runtime behavior. All reference ID matches produce confidence 1.0 regardless of match_weight, and conditions are never evaluated. These fields exist for forward compatibility.

Reference identifiers serve two purposes:

1. Deterministic matching. When two records share the same reference identifier value, they are merged immediately with full confidence, bypassing all other rules:

   # If record A has duns="123456789" and record B has duns="123456789",
   # the engine merges them with confidence 1.0 without evaluating any rules.

2. Cross-referencing. Look up canonical records by external ID through the search API:

   GET /v1/canonical/search?q=123456789&entity_type=company

Common Reference Identifier Systems

Name	Authority	Description
duns_number	dnb	Dun & Bradstreet 9-digit identifier for businesses
lei_code	gleif	Legal Entity Identifier (ISO 17442)
ein	irs	US Employer Identification Number
vat_number	eu_vies	EU VAT registration number
company_number	companies_house	UK company registration number
abn	abr	Australian Business Number
gln	gs1	Global Location Number

Complete Governance Example

A realistic spec snippet combining metadata, governance policies, exclusions, relations, and reference identifiers:

api_version: kanoniv/v1
identity_version: "3.2"

entity:
  name: company
  description: "B2B company identity across CRM, enrichment, and financial systems"

metadata:
  name: "Company Identity Resolution"
  description: "Production company identity resolution with full governance"
  owner: [email protected]
  tags: [production, b2b, company, v3]

governance:
  require_freshness: true
  require_schema: true
  required_tags:
    - production

exclusions:
  - type: field_value
    field: company_name
    operator: "eq"
    value: "Test Company"
  - type: field_value
    field: company_name
    operator: "eq"
    value: "DO NOT USE"
  - type: field_value
    field: email_domain
    operator: "contains"
    value: "test"
  - type: field_value
    field: status
    operator: "eq"
    value: "deleted"
  - type: field_value
    field: country_code
    operator: "is_empty"
  - type: explicit_pair
    left_id_field: "company-old-abc"
    right_id_field: "company-new-xyz"
  - type: cross_tenant
    enabled: true

relations:
  - name: company_contacts
    from_entity: customer
    to_entity: company
    join_key: company_id
    cardinality: many_to_one
  - name: parent_subsidiary
    from_entity: company
    to_entity: company
    join_key: parent_company_id
    cardinality: many_to_one
    propagate_match: true
  - name: company_invoices
    from_entity: invoice
    to_entity: company
    join_key: billing_company_id
    cardinality: many_to_one

reference_identifiers:
  - name: duns_number
    type: external
    authority: dnb
    field: duns
    match_weight: 1.0
  - name: lei_code
    type: external
    authority: gleif
    field: lei
    match_weight: 1.0
  - name: tax_id
    type: external
    authority: irs
    field: ein
    match_weight: 0.9

sources:
  salesforce:
    adapter: postgres
    location: salesforce_mirror/accounts
    primary_key: sf_account_id
    tags: [production]
    freshness:
      max_age: "12h"
    schema:
      company_name: { type: string }
      email_domain: { type: string }
      duns: { type: string, pii: false }
      country_code: { type: string }

  dnb_enrichment:
    adapter: csv
    location: data/dnb_enriched.csv
    primary_key: duns
    tags: [production]
    freshness:
      max_age: "168h"  # Weekly refresh
    schema:
      company_name: { type: string }
      duns: { type: string }
      lei: { type: string }
      revenue: { type: string }

  billing:
    adapter: postgres
    location: billing_db/companies
    primary_key: billing_id
    tags: [production]
    freshness:
      max_age: "24h"
    schema:
      company_name: { type: string }
      ein: { type: string }
      billing_status: { type: string }

rules:
  - name: duns_exact
    type: exact
    field: duns
    weight: 1.0

  - name: company_name_fuzzy
    type: similarity
    field: company_name
    algorithm: jaro_winkler
    threshold: 0.88
    weight: 0.7

  - name: domain_exact
    type: exact
    field: email_domain
    weight: 0.8

decision:
  scoring:
    method: weighted_sum
  thresholds:
    match: 0.9
    review: 0.75
    reject: 0.3
  conflict_strategy: prefer_high_confidence
  review_queue:
    enabled: true
    sla_hours: 336

survivorship:
  default: source_priority
  overrides:
    - field: revenue
      strategy: most_recent
    - field: duns
      strategy: most_complete

This example demonstrates several governance principles working together:

• Freshness enforcement ensures no source provides stale data (every source has a freshness block)
• Schema enforcement aborts the pipeline if field-presence drift is detected in any source
• Required tags ensure all sources are tagged with production
• Exclusions filter out test data, invalid records, already-processed records, known non-match pairs, and cross-tenant matches
• Relations link companies to contacts, parent companies, and invoices
• Reference identifiers enable deterministic matching and cross-referencing through DUNS, LEI, and EIN

Tier Requirements

Feature	Required Tier	Description
Metadata (name, description, owner, tags)	All	Available to all tiers
Reference identifiers	Cloud	External identifier system mappings
Governance policies	Cloud	Freshness, schema, and tag enforcement
Exclusions	Cloud	Record-level and pair-level filtering
Relations	Cloud	Cross-entity relationship definitions