Databricks Data Contracts — Enforce Before You Publish

A pipeline can finish successfully while breaking the downstream assumptions that matter. A column changes type, an identifier becomes nullable, or a producer starts emitting duplicate business keys. The table still exists — the data product is no longer dependable.

At NexOps, we keep self-service and governance together. Our verified work at RIVM includes an Azure national public health data platform, Unity Catalog governance and a self-service data onboarding platform. That context informs our approach here; it is not a claim that this enforcement pattern was deployed there.

We turn a data contract into a publication gate on Databricks. We separate schema checks, row-level constraints and operational obligations, then show a batch implementation with explicit environment testing. Our position is straightforward: a contract is useful only when a failed obligation changes what the platform allows to happen.

The Problem

A documented schema is not an enforced contract. It describes fields, but usually leaves unanswered who can change them, which failures block publication and what happens when consumers depend on yesterday's behaviour.

This is an engineering pattern, not a new Databricks feature. We combine explicit Spark validation, enforced Delta table constraints and governed access to make publication a deliberate decision.

We separate a contract into three layers. Each needs an explicit enforcement point rather than a promise buried in documentation.

Delta constraints provide defence in depth for row-level invariants. According to the Databricks constraints documentation, NOT NULL and CHECK constraints are enforced, and a violating transaction fails. Primary key, foreign key and unique constraints are informational — they do not enforce uniqueness or referential integrity.

These constraints do not, by themselves, establish freshness, consumer compatibility or an agreed recovery process. The example assumes provisioned Unity Catalog schemas, Delta tables and a Spark execution context, plus identities with appropriate access.

Why It Matters for Platform Teams

Publication is a separate responsibility from ingestion. Landing data proves that the platform received something; publication means that consumers can depend on it. Treating those as one operation makes every upstream surprise a downstream incident.

Our preferred architecture is landing data, a restricted candidate snapshot, validation, then publication to a consumer-facing table. The snapshot gives validation and publication the same input, while keeping rejected candidates outside the normal consumption path.

The access boundary matters as much as the checks. Unity Catalog permissions should separate consumer access to reporting data from operator access to landing and staging data. A staging schema is not private merely because its name contains “staging”.

We also distinguish correctness from timeliness. Blocking a malformed batch protects correctness, but can leave consumers reading an older valid batch. The contract must say how that stale state is exposed — silence is not a recovery strategy.

For platform teams, the benefit is a repeatable decision boundary. Producers know what will be rejected, consumers know what is guaranteed, and operators know which failures require intervention.

Use Case: Governed Public Health Reporting

Starting situation. Consider a hypothetical national health institute onboarding daily regional aggregates from several reporting organisations. It has strict classification and sovereignty requirements, limited engineering capacity and analysts who need reproducible reporting. Submissions flow from restricted landing tables into shared analytical tables, but each producer currently follows a different validation process.

Why the usual approach breaks. Each ingestion pipeline performs its own cleaning, often casting unexpected types or replacing missing values. These transformations can conceal a producer change rather than resolve it. Analysts discover the discrepancy later, after different teams have applied different interpretations.

Applying the change. The ingestion job materialises each closed submission into a candidate table that must remain unchanged during validation and publication. Validation checks its schema, permitted status values, non-negative counts and within-batch business-key uniqueness before appending it to the reporting table. Permissions keep candidates away from ordinary consumers, while target-table constraints provide another boundary against invalid writes.

What to measure. Track rejected batches by rule, time from receipt to publication, age of the latest accepted reporting date and time to resolve producer violations. Record downstream reconciliations and correction requests as well. Success means fewer surprises and clearer recovery, not simply fewer failed jobs.

These are illustrative contract rules, not a specification for an actual public health dataset. Classification, disclosure controls and permitted uses remain separate governance requirements.

Implementation

1. Define the publication agreement first

Start with one dataset and named producer and consumer owners. Agree the business key, allowed changes, blocking rules and correction process before building a generic contract framework.

For this example, the logical key is region and reporting date. Counts must be non-negative, status must be either provisional or final, and a candidate batch cannot repeat that key. Across-batch corrections require a separate publication strategy — an append alone does not solve them.

Keep the specification, validation code and deployment configuration in the same reviewed repository. If they are maintained separately, require a compatibility check between them.

Define identifier semantics precisely rather than saying “not blank”. Here we reject null, empty and space-only region codes; we do not claim that this checks every whitespace character or validates membership in an approved region register.

2. Put durable row rules on the target table

Create the target in a provisioned reporting schema, then add its check constraints. Complete this setup before granting the publication identity access to write data.

CREATE TABLE health_dev.reporting.regional_daily_cases (
  region_code STRING NOT NULL,
  report_date DATE NOT NULL,
  case_count BIGINT NOT NULL,
  status STRING NOT NULL
)
USING DELTA;

-- Default trimming removes spaces, not every whitespace character.
ALTER TABLE health_dev.reporting.regional_daily_cases
ADD CONSTRAINT region_code_present
CHECK (length(trim(region_code)) > 0);

ALTER TABLE health_dev.reporting.regional_daily_cases
ADD CONSTRAINT case_count_non_negative
CHECK (case_count >= 0);

-- These values belong to this example's business contract.
ALTER TABLE health_dev.reporting.regional_daily_cases
ADD CONSTRAINT status_allowed
CHECK (status IN ('provisional', 'final'));

Nullability and check constraints protect the target even if a writer bypasses the main validation job. The identifier expression rejects empty and space-only strings — tabs and newlines are not covered by default trimming. If the contract forbids those characters, define and test that additional rule explicitly.

This is a new-table example, not an upgrade script. Databricks checks existing rows before adding NOT NULL or CHECK constraints, so an existing dataset may require remediation first.

Adding a constraint upgrades the table writer protocol if its current writer version is below 3. Check compatibility with external Delta writers before deployment, and do not mistake these row constraints for business-key uniqueness enforcement.

3. Validate a stable candidate, then publish

The example below expects one closed batch in the landing table. It rejects unexpected columns and types rather than quietly casting them into the expected interface.

from uuid import uuid4

from pyspark.sql import functions as F

# Schemas are pre-provisioned; names come from trusted deployment configuration.
source_table = "health_dev.landing.regional_daily_cases_batch"
target_table = "health_dev.reporting.regional_daily_cases"
candidate_table = f"health_dev.onboarding.candidate_{uuid4().hex}"

expected_types = {
    "region_code": "string",
    "report_date": "date",
    "case_count": "bigint",
    "status": "string",
}

incoming = spark.table(source_table)

# Comparing names and types rejects additions, removals and type changes.
# Column order is not part of this example's contract.
if (
    len(incoming.columns) != len(expected_types)
    or dict(incoming.dtypes) != expected_types
):
    raise ValueError("Contract failure: unexpected schema")

# Materialise once so validation and publication use the same candidate.
incoming.select(*expected_types.keys()).write.format("delta").mode(
    "errorifexists"
).saveAsTable(candidate_table)

candidate = spark.table(candidate_table)

invalid_row = (
    F.col("region_code").isNull()
    # Match the target rule: reject empty and space-only identifiers.
    | (F.length(F.trim(F.col("region_code"))) == 0)
    | F.col("report_date").isNull()
    | F.col("case_count").isNull()
    | (F.col("case_count") < 0)
    | F.col("status").isNull()
    | (~F.col("status").isin("provisional", "final"))
)

# Raise without logging payloads: diagnostic logs may have broader access.
if candidate.where(invalid_row).limit(1).count():
    raise ValueError("Contract failure: invalid row")

duplicate_keys = (
    candidate.groupBy("region_code", "report_date")
    .count()
    .where(F.col("count") > 1)
)

if duplicate_keys.limit(1).count():
    raise ValueError("Contract failure: duplicate key within candidate")

# Append is illustrative; retries and corrections need an explicit design.
candidate.select(*expected_types.keys()).write.format("delta").mode(
    "append"
).saveAsTable(target_table)

Spark documents DataFrame.columns as a list of column names and DataFrame.dtypes as name–type pairs. The comparison deliberately ignores column order but checks the exact top-level type strings. It does not compare schema nullability metadata — required values are checked separately.

The explicit null checks prevent missing values from escaping the invalid-row predicate through SQL null semantics. Column.isin expresses membership in the permitted status values, while grouping and counting identifies repeated keys within this candidate.

The materialised candidate avoids validating one evaluation of the source and publishing another. Materialisation alone does not make the table immutable — access controls and job coordination must prevent changes between validation and publication. Its lifecycle also needs a cleanup policy.

This is deliberately not an exactly-once implementation. A retry after a successful append can duplicate data, and different batches can contain the same key. Before production, choose a correction and idempotency strategy, such as controlled key-based replacement with retained submission history and coordinated writers.

An empty candidate also passes these checks. If the agreement requires a non-empty submission, or an explicit “no cases” record, add that rule rather than treating the absence of invalid rows as proof of completeness.

4. Promote the contract through DTAP

Keep the rules consistent across DTAP — development, test, acceptance and production. Change catalogue names, identities and fixtures through deployment configuration, not by commenting out checks.

Use our declarative CI/CD approach for Databricks bundles to package the job and environment configuration. The official bundle jobs tutorial describes validating, deploying and running jobs with Declarative Automation Bundles.

The contract gate belongs inside the deployed workload — bundle configuration validation does not validate tomorrow's data. Run tests against the target execution environment, particularly when changing runtime versions or dependencies.

Include fixtures for missing and extra columns, incorrect types, nulls, negative counts, invalid statuses and duplicate keys. Test empty strings, spaces, tabs and newlines separately so the identifier rule's actual scope remains visible.

Tests should assert that the reporting table remains unchanged after a rejected candidate. A test that merely observes an exception misses accidental publication before the failure.

5. Record decisions without exposing payloads

For each candidate, record its submission identifier, contract version, validation outcome and publication outcome. Add non-sensitive rule identifiers and timestamps so operators can distinguish a rejected batch from a failed publication.

Treat these records as an audit trail, not as a substitute for transactional publication. Delta's transaction guarantees for a table do not make separate candidate, reporting and decision-record writes one atomic operation. Design recovery for the case where data publication succeeds but recording its outcome fails.

Agree who receives alerts and who can authorise a correction. Retain rejected data only when classification and retention policies permit it.

Review effective access using the Unity Catalog privilege-management guidance. Include inherited access and ownership in that review — restricting one table is insufficient if broader permissions still allow unwanted access or changes.

Pitfalls and When Not to Use It

Automatic coercion can hide a broken interface. Casting a newly supplied string into a number may restore the physical type while concealing changed semantics. Allow transformations only when the contract explicitly defines them.

Strict schema matching is a choice, not a universal rule. Our example rejects additive columns. If consumers permit compatible additions, encode that policy explicitly and test it — do not rely on whichever writer behaviour happens to be configured.

Passing row checks does not prove completeness. A batch can contain valid records while omitting expected regions or arriving late. Coverage and delivery obligations need their own checks and escalation paths.

Validation has a compute cost. Materialising candidates, scanning for invalid rows and grouping for duplicates all consume resources. Measure validation duration and compute consumption against the harm of publishing invalid data, then consolidate checks where appropriate.

A gate can preserve bad assumptions. A non-negative count may be valid for this example but inappropriate for adjustment records. Business owners must approve semantics; engineers should not invent them from column names.

We would not start with a contract framework for disposable exploratory data with no stable consumers. We would also avoid applying this batch pattern unchanged to a low-latency stream: publication boundaries, state, late arrivals and failure handling need a different design.

For a shared reporting product, however, we would adopt a narrow gate early. A small, enforceable agreement is more valuable than an extensive specification that every writer can bypass.

Key Takeaways

1. Separate ingestion from publication. Consumers should read accepted data, not whatever most recently arrived.
2. Define the whole agreement. Schema, row rules, ownership, freshness and corrections need explicit decisions.
3. Validate a stable candidate. Use the same unchanged input for checking and publication.
4. Know what constraints enforce. NOT NULL and CHECK protect row invariants; informational keys do not enforce uniqueness.
5. Test failure and recovery through DTAP. Verify unchanged targets, identifier edge cases, retries, corrections and stale-data signalling.
6. Start small and measure the cost. Enforce the rules consumers actually depend on before building a general framework.