Paying for Spark on Small Data? Choose the Engine Without Rewriting Your Rules

Picture a nightly job that reads one CSV of customer records, a few hundred megabytes, checks that status is a known value and revenue is not negative, and writes the result. It runs on Spark because everything else does. Much of its run is spent waiting for compute, and the same two rules also live in a pandas script someone wrote for local testing and in a dbt test further down the line.

This happens because the engine was chosen once, for the largest workload, and every smaller job inherited it. When a team does move a small job to a lighter engine, the rules usually get rewritten in that engine's API, and now there are two copies to keep in step.

This article covers the realistic ways to stop paying for distributed compute you do not need, where each fits, and one way to keep a single definition of the rules when different jobs run on different engines. It does not quote cost savings or benchmarks: those depend on your data, your platform and how you pay for compute, so measure them on your own workload.

Why small jobs end up on Spark

Spark is designed to split work across machines. For a dataset that fits comfortably on one machine, that design brings overhead you still pay for: a JVM, a cluster or session to start, and a scheduler coordinating work a single process could do. Single-node engines such as Polars and DuckDB run inside a Python process and start without a cluster.

The engine is rarely the hard part of switching. The hard part is everything written against it: the validation, the casting, the merge logic. That is what makes a second engine expensive to adopt.

The ways teams solve it

All of these work. They solve different halves of the problem.

Stay on Spark, but with less overhead. Databricks serverless compute removes cluster provisioning and idle time for notebooks and jobs, and Photon is Databricks' native vectorised engine. If your stack is Databricks end to end, this may be all you need: no rule rewrite, just a different compute setting. Spark Connect (Spark 3.4 and later) separates a thin client from the Spark server, which helps development, but a Spark server still executes the plan.

Run the small jobs on Polars or DuckDB directly. This is the simplest way off the cluster for data that fits on one machine. The cost is that the checks are now written in Polars expressions or DuckDB SQL, separately from the Spark version used by the large jobs.

Use one abstraction over several engines. dbt data tests are SQL queries compiled and run through an adapter, and adapters exist for Spark, Databricks and DuckDB among others. Ibis gives one Python dataframe API over Polars, DuckDB, PySpark and more; its own support matrix notes that coverage of the API varies by backend. SQLGlot transpiles SQL between dialects, and its documentation is candid that some dialect pairs do not handle every input.

Share the tables. Delta Lake and Apache Iceberg let different engines read and write the same tables; delta-rs brings Delta to Python without a JVM. This settles where the data lives. It says nothing about which rules the data must pass on the way in.

The gap: one definition of the rules

Each option above handles either the compute or the expression language. What is left is a single place that says: these are the fields and types, these are the row rules, this is what happens to a failing row, and this is how the output is written. If that definition belongs to one engine's code, choosing a second engine means copying it.

LakeLogic, an Apache 2.0 Python library, takes that definition out of the engine. A YAML data contract declares the schema and quality rules (and materialisation, when you need it), and you pick the engine when you run it. The open-source runtime has adapters for Polars, DuckDB and Spark, plus Snowflake and BigQuery for table sources. Polars and DuckDB come with the base install; Spark is the [spark] extra.

A worked example

The contract for the customer job from the opening:

version: 1.0.0
info:
  title: Customers
  table_name: customers
model:
  fields:
    - name: customer_id
      type: string
      required: true
    - name: email
      type: string
    - name: status
      type: string
    - name: revenue
      type: double
quality:
  row_rules:
    - name: revenue_non_negative
      sql: "revenue >= 0"
    - name: known_status
      sql: "status IN ('active', 'churned', 'trial')"

The input, customers.csv, has five rows: three valid, one with an unknown status (pending) and one with negative revenue. The script runs the same contract on Polars and on DuckDB:

import polars as pl
import lakelogic as ll

df = pl.read_csv("customers.csv")
for engine in ("polars", "duckdb"):
    good, bad = ll.DataProcessor("customers.yaml", engine=engine).run(df)
    print(engine, "good:", len(good), "bad:", len(bad))

polars good: 3 bad: 2
duckdb good: 3 bad: 2

Each quarantined row carries the rule it failed in a _lakelogic_errors column: c4 failed known_status and c5 failed revenue_non_negative. The CLI does the same from the file:

pip install lakelogic
lakelogic run --contract customers.yaml --source customers.csv --engine duckdb

The large jobs keep running on Spark with the same file. The line below was not run for this article (it needs pip install "lakelogic[spark]" and a Spark session), so treat it as illustrative:

good, bad = ll.DataProcessor("customers.yaml", engine="spark").run(spark_df)

When product adds a new status, you change one line in customers.yaml instead of a Polars expression, a Spark filter and a dbt test. Matching counts on five rows is a smoke test, not proof that every rule, type and file format agrees across engines. Our platform article on checking that a local pass still holds on Spark covers the cross-engine conformance cases that test that.

Choosing the engine per job

There is no universal size threshold, so this article does not give one. Ask instead:

• Does the job's data fit on one machine with headroom? If yes, try Polars or DuckDB and time it against your current Spark run.
• Does it need a catalog that Spark reaches more easily? Unity Catalog or Fabric tables are usually simpler from Spark.
• Does it need a feature only one engine supports? Check the capability matrix for sources, outputs and materialisation per engine.

When you do not need this

• You run one engine. If every job is Spark, or every job is Polars, there is no second copy of the rules to keep in step.
• Spark is already cheap enough for you. If serverless compute or a small shared cluster makes small jobs acceptable, changing engines may not be worth the effort.
• Your stack is SQL-only. dbt tests in one warehouse already give you one definition of the checks.

Limits

• Engine support is per feature. File formats, table sources and materialisation differ by engine; Spark, for example, is the recommended path for lakehouse catalog tables. Read the capability matrix for what you use rather than assuming parity.
• Engines can disagree. Type widths, file readers and expression semantics differ. Run the same test cases on each engine you deploy to.
• Local is not the platform. Permissions, catalogs and runtime versions only exist there; keep a platform run before release.
• Platform responsibilities stay put. LakeLogic runs the contract; storage, compute, access control and orchestration remain your platform's job.

Key takeaways

1. Pick the engine per job by data size and catalog needs, not by what the biggest job uses.
2. Serverless Spark and Spark Connect reduce cluster friction but still run Spark.
3. Shared tables (Delta, Iceberg) share data; they do not share rules.
4. Keep schema and rules in one definition so a second engine does not mean a second copy.
5. Test each rule on each engine you run it on, and measure speed and cost on your own workload.