Declarative Automation Bundles: A Practical Guide to CI/CD in Databricks

Many Databricks teams start with notebooks in a shared workspace, jobs configured through the UI and a growing collection of deployment scripts. That approach gets the first workload running — but leaves the deployment contract scattered across code, workspace settings and someone's memory.

Adding test and production environments exposes the gaps. Notebook changes and job configuration can travel separately, while reviewers struggle to establish which settings belong to which release.

Declarative Automation Bundles (DABs) — previously called Databricks Asset Bundles — put project configuration alongside source files under version control. We use them to make Databricks deployments reviewable and repeatable — not to replace authentication design, data-quality testing or production approvals.

The Problem

The Databricks bundle CLI reference describes bundles as a way to express projects as code and programmatically validate, deploy and run workflows, including jobs and Lakeflow pipelines. The missing piece in an ad-hoc deployment is often not another script, but a shared contract between source code, resource configuration and the target workspace.

A bundle makes that contract visible in Git. Reviewers can inspect the intended resource definitions alongside the application changes rather than reconstructing them from workspace screenshots.

That is infrastructure as code for the project's Databricks resources, not a complete platform provisioning strategy. We still need explicit boundaries for workspace infrastructure, identities, networking and data access.

Why It Matters for Platform Teams

Environment targets let a project deploy to multiple workspaces without maintaining an independent deployment script for each one. We favour a common project definition with deliberate target-specific settings over copied configuration that gradually diverges.

Shared templates give platform teams a practical starting point for standardisation. Source layout, deployment paths and identity expectations can be reviewed centrally while product teams retain ownership of pipeline logic.

Deployment ownership matters as much as configuration. Bundles track managed resources by their IDs in workspace state — display names do not establish ownership, and two deployments with the same bundle identity can interfere with one another.

Versioned configuration also improves the evidence available to auditors. It does not prove compliance or correct outputs; we still need access checks, integration tests and a traceable connection between a reviewed commit and the deployed release.

Use Case: Standardising Enterprise Pipeline Deployments

Consider a hypothetical large enterprise with several product teams shipping pipelines to a shared Databricks platform. Operational events pass through bronze ingestion and silver transformation before feeding downstream analytics, while security teams require evidence of approved deployment identities and resource ownership.

The usual approach breaks when teams promote notebooks separately from UI-managed jobs. A test environment may contain the right notebook code but different compute settings, paths or permissions, making a successful test poor evidence for production readiness.

We would introduce a shared bundle template containing the source layout, job definitions, environment targets and CI validation steps. Each product team would own its transformations; the platform team would review the deployment contract and require execution tests before promotion.

Measure failed promotions, manual deployment interventions and configuration discrepancies discovered during release checks. Also verify that each release maps to a reviewed commit and that representative operational events produce the expected silver records — deployment success alone proves neither.

For a large enterprise client, we delivered a Databricks data platform, adoption of Databricks Asset Bundles, SAT deployment and Terraform inner sourcing across teams. The scenario above is hypothetical; it does not attribute a particular template, release gate or current feature configuration to that client.

Implementation

1. Install and authenticate the Databricks CLI

Use the official CLI installation instructions for your operating system. That page covers the CLI installation family starting at version 0.205.0; it does not establish the minimum version for every bundle capability.

# Official installer for Linux, macOS or Windows Subsystem for Linux.
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

# Record the installed version before testing the bundle.
databricks version

The installer downloads the latest CLI release, so review the script before executing it. For CI, we would use an approved CLI version and retain the version output with the build evidence rather than silently changing tooling on every run.

Configure workspace authentication before validating or deploying. Developers and CI agents need appropriate credentials, network access and deployment permissions; a working CLI installation establishes none of those.

2. Define a small job contract

Start with a job whose execution we can test end to end. This example uses a single notebook entry point for ingestion and transformation, avoiding an unverified assumption about multi-task dependency conditions.

Define the bundle in databricks.yml and add the notebook at src/enterprise_operations.py. The serverless environment structure below follows the supplied SAT bundle templates; confirm that this compute configuration is available and suitable in each target workspace.

bundle:
  name: enterprise-operations-pipeline

resources:
  jobs:
    daily_etl:
      name: "Enterprise Operations ETL"
      tasks:
        - task_key: process_operational_events
          # One entry point owns the ingestion-to-transformation sequence.
          notebook_task:
            notebook_path: ./src/enterprise_operations.py
          environment_key: default

      environments:
        - environment_key: default
          spec:
            # The supplied SAT templates use serverless Environment v5.
            client: "5"

The notebook must implement and test the intended processing sequence; the bundle does not supply that application logic. We deliberately omit a schedule and custom dependencies until their settings have been checked against the target workload.

For classic compute, the supplied SAT templates place job_clusters inside the individual job and reference them from tasks using job_cluster_key. Runtime and node identifiers still require workspace-specific verification — we would not copy automatically selected values into a permanent platform standard without review.

3. Configure targets, deployment paths and run identity

Append the following targets to databricks.yml, replacing the illustrative hosts and service-principal application ID. This is a deployment example, not a complete permissions or authentication configuration.

targets:
  dev:
    default: true
    workspace:
      # Replace with the actual development workspace URL.
      host: https://adb-1234567890123456.7.azuredatabricks.net

  production:
    mode: production
    workspace:
      # Replace with the actual production workspace URL.
      host: https://adb-2345678901234567.8.azuredatabricks.net
      # Keep this reviewed path stable and distinct from other deployments.
      root_path: /Workspace/Applications/${bundle.name}/${bundle.target}
    run_as:
      # Replace with the intended workflow service principal's application ID.
      service_principal_name: "00000000-0000-0000-0000-000000000000"

The production run_as mapping makes the intended workflow identity explicit. Review deployment authentication and data-access permissions separately, and require execution tests under the intended workflow identity before promotion.

The CLI documentation identifies workspace.root_path as the basis of a bundle's unique deployment identity. Keep the production path stable, assign a deployment owner and prevent unrelated bundles from sharing it.

Keep credentials out of source files and YAML. Use an approved secrets-management mechanism, following the Azure Key Vault documentation and the relevant Databricks integration guidance; do not assume that ordinary configuration substitution retrieves runtime secrets.

4. Validate, deploy and test through CI

A few CLI commands do not constitute a production-ready CI pipeline. Repository checkout, an approved CLI installation, authentication, network access, deployment permissions and production approvals remain external prerequisites.

set -euo pipefail

# Run inside the bundle folder and inspect the reported deployment identity.
databricks bundle validate

# Deploy explicitly to development only after validation succeeds.
databricks bundle deploy --target dev

Validation provides a configuration gate, while deployment creates or updates bundle-managed resources. Run the deployed workflow and check its outputs separately — neither command proves that ingestion or transformation is correct.

Promote the same reviewed source through your environments, repeating target-specific checks before deployment. Retain the commit identifier, CLI version, validation output, deployment result and execution-test evidence for each release.

For a related example, see Deploying Databricks SAT with Declarative Automation Bundles Across DTAP. SAT's upgrade documentation describes bundle deployment as an advanced installation path; it is a useful deployment example, not proof that an arbitrary bundle is production-ready.

Pitfalls and When Not to Use It

Bundles fit projects with multiple contributors, repeatable deployment requirements and controlled promotion. For a disposable experiment with no promotion requirement, we would not introduce a full CI/CD framework just to deploy one notebook.

Do not treat mode: production as a security boundary. Deployment authentication, workflow identity, permissions and execution tests still need review, and target-specific behaviour must be tested rather than inferred from the mode name.

Resource deletion is part of deployment. The CLI documentation states that removing a previously deployed resource from bundle configuration removes it from the target workspace on deployment; review deletions as carefully as additions.

For an existing job or pipeline, do not assume that matching its display name adopts it into the bundle. The documented bundle deployment bind command associates an existing resource with a bundle definition, and the next deployment applies that definition — inspect the existing resource and proposed changes first.

Do not routinely bypass deployment protections. The CLI documentation reserves --force-lock for a stale lock left by an interrupted deployment, and recommends --select only for development rather than production because downstream resources are not updated.

We recommend one deployment owner per resource. If Terraform and bundles coexist, document their boundaries instead of allowing both to manage the same job or pipeline; our Azure Terraform practices cover the surrounding infrastructure discipline.

Key Takeaways

1. Version the deployment contract — Keep source files and resource definitions together so reviewers can assess the intended release.
2. Standardise with templates — Establish common deployment expectations without taking pipeline ownership away from product teams.
3. Validate and execute — Configuration validation is a gate, not a substitute for workflow and data-quality tests.
4. Make targets and identities explicit — Verify workspace access, compute suitability, deployment credentials and workflow permissions separately.
5. Protect deployment state — Keep bundle identity stable, review deletions and bind existing resources deliberately.
6. Keep ownership clear — Avoid overlapping resource management and retain evidence linking each deployment to a reviewed commit.