Databricks 16.0 — What Matters for Platform Engineers

Databricks Runtime 16.0 — The Features That Matter

Databricks Runtime 16.0 shipped with a dense changelog. Rather than rehashing every line item, here's what actually impacts your day-to-day platform engineering — and what you should prioritise adopting.

1. Liquid Clustering is GA

This is the headline feature. Liquid Clustering replaces traditional partitioning and Z-ordering with an automatic, adaptive clustering strategy.

Before (manual partitioning):

CREATE TABLE events
USING DELTA
PARTITIONED BY (date, region)
LOCATION 's3://lake/events'

After (Liquid Clustering):

CREATE TABLE events
USING DELTA
CLUSTER BY (date, region, event_type)
LOCATION 's3://lake/events'

The difference is significant:

• No more partition pruning tuning — Liquid Clustering adapts to your actual query patterns
• Add or change clustering columns without rewriting data — this was impossible with traditional partitioning
• Automatic compaction — small files are merged without manual OPTIMIZE runs

When to migrate

Migrate tables that suffer from:

• Over-partitioned directories (thousands of small partitions)
• Frequent OPTIMIZE ZORDER BY maintenance jobs
• Query patterns that span multiple partition boundaries

Don't migrate tables that are append-only with simple date-based access patterns — traditional partitioning still works fine there.

2. Predictive I/O for Reads

Predictive I/O uses ML models trained on your query history to pre-fetch data blocks before they're requested. In benchmarks, this reduces scan times by 30-40% on wide tables with complex predicates.

How to enable:

ALTER TABLE my_table SET TBLPROPERTIES ('delta.enablePredictiveIO' = 'true');

The catch: it requires Unity Catalog managed tables and at least 30 days of query history to build effective predictions. Start enabling it now so the models begin training.

3. Unity Catalog — Fine-Grained Row & Column Filters

Unity Catalog now supports row-level security and column masking natively:

-- Row filter: analysts only see their own region
ALTER TABLE sales SET ROW FILTER region_filter ON (region);

-- Column mask: PII is hashed for non-privileged users
ALTER TABLE customers ALTER COLUMN email SET MASK mask_pii;

This is a game-changer for multi-tenant platforms. Previously, you needed views, wrapper notebooks, or external policy engines. Now it's built into the catalog layer with full audit logging.

Impact for platform engineers

• Remove custom view-based security layers — Unity Catalog handles it natively
• Simplify audit compliance — all access control is in one place with full lineage
• Enable self-service analytics — business users can query tables directly without data engineering intermediaries

4. Serverless Workflows GA

Databricks Workflows now supports serverless compute for job orchestration:

# In your databricks.yml (DABs)
resources:
  jobs:
    daily_pipeline:
      tasks:
        - task_key: "transform"
          notebook_task:
            notebook_path: "./transform.py"
          environment_key: "serverless"

  environments:
    - environment_key: "serverless"
      spec:
        client: "2"

Benefits:

• Zero cold-start for orchestration — jobs start in seconds instead of minutes
• No cluster management — compute scales automatically
• Cost reduction — pay only for execution time, not idle clusters

5. AI Functions in SQL

You can now call AI models directly from SQL queries:

SELECT
  customer_id,
  feedback_text,
  ai_classify(feedback_text, ARRAY('positive', 'negative', 'neutral')) AS sentiment,
  ai_summarize(feedback_text, 50) AS summary
FROM customer_feedback
WHERE date > '2026-01-01'

This unlocks AI capabilities for SQL-first analysts without requiring Python notebooks or ML infrastructure. Under the hood, it uses Foundation Model APIs with automatic batching and rate limiting.

What to Prioritise

Here's our recommended adoption order:

Start with Unity Catalog security features — they reduce complexity immediately. Then adopt Liquid Clustering for new tables. Migrate existing tables only after validating performance on your specific workloads.

Conclusion

Runtime 16.0 is a platform engineering release. The features that matter most — Liquid Clustering, Unity Catalog security, Serverless Workflows — all reduce operational overhead while improving governance. That's the right trajectory for any serious lakehouse platform.

Running an older runtime and planning an upgrade? Talk to us about migration strategies.