Data Quality Management: Which Parts Belong in the Pipeline

Picture a data team asked to "sort out data quality" after a finance report used trips with negative fares. Someone proposes a data quality platform. Someone else says a few SQL checks in the pipeline would have caught it. Both are partly right, and the evaluation stalls because nobody has said which problem they are solving.

"Data quality management" names several jobs at once: agreeing what valid means, checking data against it, deciding what happens to failures, watching quality over time, and getting someone to fix the cause. A tool that is excellent at one of these can be weak at another, so comparing tools before splitting the jobs produces a long feature table and no decision.

This article splits data quality management into those jobs, surveys the main kinds of tools fairly, and shows one approach for the jobs that sit closest to the pipeline: rules in a versioned file, checked on every run, with failing rows kept. It does not cover master data matching, data cleansing or stewardship workflows, where dedicated platforms do work that pipeline rules do not.

The five jobs inside data quality management

The first three are tightly coupled to the pipeline: a rule that is defined in one place and checked in another tends to drift, and the decision about a failing row has to be made where the row is. Observing and remediating are more organisational, and that is where a central platform often earns its place.

Why the split matters

A common pattern is rules written three times: once in a spreadsheet or catalogue for governance, once as SQL in a transformation, and once as a monitor after the table lands. When a definition changes, one copy gets updated. Another is quality handled only by observation: the table is checked after consumers can already read it, so the alert arrives after the damage. Neither is a tooling failure. Both come from not deciding which job each tool owns.

The ways teams do it today

All of these work in the right setting, and many teams use two or three together:

Some detail. Commercial data quality suites from the large data management vendors typically combine profiling, rule authoring, cleansing, matching and steward workflows; if your problem is duplicate customer records across systems, that is their territory. Great Expectations gives you a large library of expectations and validation results; what to do with failing rows is up to your code. Soda runs checks and data contracts written in its own language, on hosted or self-hosted runners, with results managed in Soda Cloud. dbt data tests sit next to the models they check and run after the model is built. Observability tools in the style of Monte Carlo learn normal freshness and volume across many tables and alert on anomalies without hand-written rules. Databricks pipeline expectations can warn about, drop or stop on failing records inside Databricks pipelines.

The gap: define, check and act in one file

The question these leave partly open is the one closest to the pipeline: can the rule definition, the check and the decision about failing rows sit in one reviewed file, run inside the pipeline on more than one engine, and keep every failing row with the reason it failed? LakeLogic, an open-source (Apache 2.0) Python library, takes that approach. A data contract in YAML declares fields and rules; the same contract runs on Polars, DuckDB, Spark, Snowflake or BigQuery.

A worked example

Take RideFlow, a fictional ride-hailing business, where each row is one completed trip. Its trip data needs three rules: fares are never negative, payment methods come from a known list, and each trip appears once.

version: 1.0.0
info:
  title: Silver trips completed
  domain: marketplace
  system: rideflow
  owner: marketplace-data-team
model:
  fields:
    - name: trip_id
      type: string
      required: true
    - name: city_code
      type: string
    - name: fare_amount
      type: float
    - name: payment_method
      type: string
quality:
  row_rules:
    - name: fare_not_negative
      sql: fare_amount >= 0
      category: validity
    - accepted_values:
        field: payment_method
        values: [card, cash]
  dataset_rules:
    - unique: trip_id
quarantine:
  enabled: true
lineage:
  enabled: true

Run it against a small file of four trips, one with a fare of -12.50 and one paid by voucher:

from lakelogic import DataProcessor

result = DataProcessor("trips.yaml").run_source("trips.csv")
print(result.good.select(["trip_id", "fare_amount"]))
print(result.bad.select(["trip_id", "_lakelogic_errors"]))

# good: t1, t4
# bad:  t2  ["Rule failed: fare_not_negative (fare_amount >= 0)"]
#       t3  ["Rule failed: payment_method_accepted_values (...)"]

The two good trips continue; the two failing trips are held in result.bad with the rule each one broke, so someone can decide whether a negative fare is a refund or a bug. With lineage on, each output row also carries columns such as _lakelogic_run_id, _lakelogic_source and _lakelogic_processed_at, which tie it back to the run and file that produced it.

Dataset rules behave differently from row rules, and the difference matters. When we added a duplicate trip to the file, the unique rule was recorded as failed on the run, but both copies stayed in the good output because no single row is wrong. Setting quality.fail_pipeline_on_dataset_error: true made the same run stop with an error instead. Choose that per table: holding a row and stopping a load are different decisions.

From the command line the same contract runs with lakelogic run --contract trips.yaml --source trips.csv --output-bad quarantine.parquet, and lakelogic bootstrap can draft starting contracts from existing landing files for you to review.

When you do not need this

• Your models already live in dbt. dbt tests and model contracts may cover the checks you need without another tool.
• One platform, simple rules. Platform expectations are enough if you do not need portable rules or kept failing rows.
• The problem is unknown unknowns. If you cannot write the rule yet, anomaly detection from an observability tool finds what you did not think to check.
• The problem is master data. Matching and merging records across systems needs a data quality or MDM platform.

Limits worth knowing

• A contract checks only what it declares. A plausible but wrong value passes.
• LakeLogic Core checks and records per run. Trend views, alert routing and remediation tracking need monitoring and ticketing around it.
• Quarantine holds rows; someone still has to review and reprocess them.
• It does not replace your catalogue or access control. A contract describes valid data, not who may read it.

Observing quality over time is its own topic. Our article on data product quality metrics covers which numbers to track and why a single score misleads.

Key takeaways

1. Split the jobs first. Define, check, act, observe and remediate are different problems; choose a tool per job.
2. Keep define, check and act together. Rules defined in one place and run in another drift apart.
3. Decide row versus batch handling. Hold failing rows, and choose explicitly which dataset failures stop a load.
4. Use a platform where it does unique work. Matching, cleansing and stewardship are worth paying for if you have those problems.