Skip to content

ADR-0025 — Source-Acquired Canonical Dataset Lineage v2

Status: ACCEPTED Date: 2026-09-04

Context

dataset-manifest-v1 requires every non-raw dataset to declare at least one DatasetIdentity parent and a transform. A source/archive acquisition is not a DatasetIdentity, so a direct source/archive → canonical materialization cannot be represented faithfully in v1. Creating a raw proxy, fake parent, or source provenance subsystem would violate the frozen data-plane boundaries.

Decision

Introduce the explicitly versioned dataset-manifest-v2. Version 1 remains frozen and is never reinterpreted.

Dataset topology is:

raw:
  origin absent, derived_from absent, transform absent

canonical + dataset_derived:
  origin=dataset_derived, derived_from >= 1, transform required

canonical + source_acquired:
  origin=source_acquired, derived_from absent, transform required

features:
  origin=dataset_derived, derived_from >= 1, transform required

origin describes dataset topology only. It does not identify a venue, source, archive, storage location, source semantics, mapping, or coverage completeness, and it is not part of DatasetIdentity or the catalog natural key.

For v2, transform means the semantic transformation that produced the dataset. It is persisted on catalog.dataset_lineage only for datasets with declared dataset parents. A source-acquired canonical dataset retains the transform in its durable manifest and has exactly zero lineage rows; no synthetic parent or catalog transform column is introduced.

Source provenance remains owned by coverage-manifest-v1 acquisition, assertions, and evidence. Coverage v1 is unchanged. S14 performs no additional origin-to-coverage cross-check beyond the existing S13 quality-report and coverage-hash binding.

Manifest validators and S14 use explicit schema-version dispatch. Unsupported versions fail closed. V1 raw, canonical, features, and zero-parent rejection retain their existing semantics.

S13 semantics, the catalog DDL, partition and trade contracts, DataGateway, materializer, and Golden fixture remain unchanged. The existing catalog tables represent a source-acquired canonical dataset as one catalog.datasets row and zero catalog.dataset_lineage rows.

The v2 manifest is sufficient for a future semantic rebuild to reconstruct that same zero-lineage topology. No catalog rebuild runtime is introduced by this decision.

Consequences

  • Direct source/archive → canonical publication is representable without fake lineage.
  • Existing v1 manifests remain compatible and retain their frozen meaning.
  • Generic S14 parent resolution must understand v1 and v2 topology, while its exact-lineage and Phase-5 verification algorithms remain unchanged.
  • Source-specific acquisition evidence remains outside the generic manifest and publication contracts.
  • The implementation remains a narrow contract/runtime compatibility slice; source registries, artifact tables, provenance frameworks, new layers, and catalog migrations are out of scope.