Open-source data contracts

Reliable pipelines,
defined in YAML.

Define schemas, quality rules and materialization once. Validate locally, check contract changes in CI, and run governed pipelines with Polars, DuckDB or Spark.

Apache 2.0 · Python 3.10+ · Polars · DuckDB · Spark
Run the 5-minute quickstart View on GitHub
PyPI …
orders_contract.yaml 
# Business expectations as code
model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: amount
      type: float
      required: true
quality:
  row_rules:
    - name: positive_amount
      sql: "amount > 0"
100Input
97Accepted
3Quarantined

Stop rebuilding the same controls in every pipeline. LakeLogic turns version-controlled contracts into executable validation, quarantine and materialization behaviour while complex transformations remain normal Python or SQL.

Start with the base package

Polars and DuckDB are included. Add the Spark extra when you need distributed lakehouse execution.

pip install lakelogic
pip install "lakelogic[spark]"
One contract, three control points

Move feedback closer to the change

01 · Develop

Validate representative data locally

Run schemas and quality rules with Polars or DuckDB before paying the startup cost of a remote Spark environment.

02 · Review

Check contract changes in CI

Use configured gates to identify breaking schemas, missing PII declarations and lineage changes before deployment. Some gates require comparison context.

03 · Execute

Retain failures with evidence

Separate accepted and failed records, preserve diagnostic context, and stop downstream work when dataset-level thresholds are breached.

Engine boundaries matter. The contract model is shared, but engines and catalog combinations are not identical. Review the capability matrix before choosing Spark, Polars, DuckDB, Delta or Iceberg.

See the pattern in a
working data-mesh example

RideFlow on Databricks

RideFlow is a fictional business used to demonstrate domain-owned data products on Databricks Unity Catalog. The repository shows contracts, Bronze–Silver–Gold flows, quarantine, lineage, SCD Type 2 and cross-domain dependencies.

  • Inspect real YAML contracts and deployment definitions.
  • Trace accepted and quarantined outputs through the pipeline.
  • See where LakeLogic stops and Databricks infrastructure takes over.
Explore the reference implementation
Reference architecture

Domain autonomy with shared controls

Independent domain products use common contract machinery without claiming that YAML alone creates organisational ownership or platform governance.

Bronze Silver Gold
6Domains
3Layers
View architecture and code →
Faster onboarding

Infer a draft, then review it

infer_contract() can inspect a supported file or table sample and return a draft with fields, types and suggested rules. The draft remains reviewable YAML—not an invisible production decision.

PII detection and AI-assisted descriptions are optional capabilities with additional dependencies or configured providers.

draft_contract.py 
from lakelogic import infer_contract

draft = infer_contract(
    "data/orders.csv",
    title="Orders Bronze",
    suggest_rules=True,
)

draft.show()
draft.save("contracts/orders.yaml")
Where LakeLogic fits

Start with pipelines.
Expand with evidence.

01

Reliable data products

Put schemas, quality rules, ownership metadata and service expectations beside the code that produces the dataset.

02

Domain-owned lakehouses

Reuse contract machinery across teams while domains retain responsibility for business meaning and product changes.

03

Deterministic AI boundaries

Let AI assist with drafts or proposed changes, then use explicit contracts, review and runtime gates to control what executes.

Documentation

Choose an engine
and build a contract

 Apache 2.0

Read the source
and run the tests
From the blog

Go deeper on the design choices

Read all posts →
Architecture

Stop the Spark Tax: One Data Contract, Any Engine

How shared contract intent can reduce logic drift while respecting engine-specific boundaries.

Read → Reliability

How Quarantine Saved Our Pipeline

Retain failed records with evidence instead of silently discarding them or losing the whole run.

Read → Contracts

Data Contracts vs Schema Validation

Schema checks shape. Contracts can also express quality, ownership, service expectations and outputs.

Read →

Run the contract before you trust the claim.

Open the quickstart, inspect the accepted and failed records, then decide whether LakeLogic fits your pipeline.

Run in Google Colab View the repository