Declarative Automation Bundles: A Practical Guide to CI/CD in Databricks

Many Databricks teams start with notebooks in a shared workspace, jobs configured through the UI and a growing collection of deployment scripts. That approach gets the first workload running, but leaves the deployment contract scattered across code, workspace settings and someone's memory.

Adding test and production environments exposes the gaps. Notebook changes and job configuration can travel separately, while reviewers struggle to establish which settings belong to which release.

Declarative Automation Bundles (DABs), previously called Databricks Asset Bundles, put project configuration alongside source files under version control. To prevent deployment drift, we recommend bundles: version the deployment contract and test each target before promotion.

The Problem

The Databricks bundle CLI reference describes bundles as a way to express projects as code and programmatically validate, deploy and run workflows, including jobs and Lakeflow pipelines. The missing piece in an ad-hoc deployment is often not another script, but a shared contract between source code, resource configuration and the target workspace.

That is infrastructure as code for the project's Databricks resources, not a complete platform provisioning strategy. We still need explicit boundaries for workspace infrastructure, identities, networking and data access.

Why It Matters for Platform Teams

Environment targets let a project deploy to multiple workspaces without maintaining an independent deployment script for each one. We favour a common project definition with deliberate target-specific settings over copied configuration that gradually diverges.

Shared templates give platform teams a practical starting point for standardisation. Source layout, deployment paths and identity expectations can be reviewed centrally while product teams retain ownership of pipeline logic.

Deployment ownership matters as much as configuration. Bundles track managed resources by their IDs in workspace state. Display names do not establish ownership, and two deployments with the same bundle identity can interfere with one another.

Versioned configuration also improves the evidence available to auditors. It does not prove compliance or correct outputs; we still need access checks, integration tests and a traceable connection between a reviewed commit and the deployed release.

Use Case: Standardising Enterprise Pipeline Deployments

Consider a hypothetical large enterprise with several product teams shipping pipelines to a shared Databricks platform. Operational events pass through bronze ingestion and silver transformation before feeding downstream analytics, while security teams require evidence of approved deployment identities and resource ownership.

The usual approach breaks when teams promote notebooks separately from UI-managed jobs. A test environment may contain the right notebook code but different compute settings, paths or permissions, making a successful test poor evidence for production readiness.

We would introduce a shared bundle template containing the source layout, job definitions, environment targets and CI validation steps. Each product team would own its transformations; the platform team would review the deployment contract and require execution tests before promotion.

Measure failed promotions, manual deployment interventions and configuration discrepancies discovered during release checks. Also verify that each release maps to a reviewed commit and that representative operational events produce the expected silver records.

For a large enterprise client, we delivered a Databricks data platform, adoption of Databricks Asset Bundles, SAT deployment and Terraform inner sourcing across teams. The scenario above is hypothetical; it does not attribute a particular template, release gate or current feature configuration to that client.

Implementation

1. Install and authenticate the Databricks CLI

Use the official CLI installation instructions for your operating system. That page covers the CLI installation family starting at version 0.205.0; it does not establish the minimum version for every bundle capability.

# Official installer for Linux, macOS or Windows Subsystem for Linux.
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
 
# Record the installed version before testing the bundle.
databricks version

The installer downloads the latest CLI release, so review the script before executing it. For CI, we would use an approved CLI version and retain the version output with the build evidence rather than silently changing tooling on every run.

Configure workspace authentication before validating or deploying. Use workload identity federation for CI agents and private network paths to the workspace. Grant deployment access through Entra ID security groups managed in Terraform.

2. Define a small job contract

Start with a job whose execution we can test end to end. This example uses a single notebook entry point for ingestion and transformation, avoiding an unverified assumption about multi-task dependency conditions.

Define the bundle in databricks.yml and add the notebook at src/enterprise_operations.py. The serverless environment structure below follows the supplied SAT bundle templates; confirm that this compute configuration is available and suitable in each target workspace.

bundle:
  name: enterprise-operations-pipeline
 
resources:
  jobs:
    daily_etl:
      name: "Enterprise Operations ETL"
      tasks:
        - task_key: process_operational_events
          # One entry point owns the ingestion-to-transformation sequence.
          notebook_task:
            notebook_path: ./src/enterprise_operations.py
          environment_key: default
 
      environments:
        - environment_key: default
          spec:
            # The supplied SAT templates use serverless Environment v5.
            client: "5"

The notebook must implement and test the intended processing sequence; the bundle does not supply that application logic. We deliberately omit a schedule and custom dependencies until their settings have been checked against the target workload.

For classic compute, the supplied SAT templates place job_clusters inside the individual job and reference them from tasks using job_cluster_key. Runtime and node identifiers still require workspace-specific verification. We would not copy automatically selected values into a permanent platform standard without review.

3. Configure targets, deployment paths and run identity

Append the following targets to databricks.yml, replacing the illustrative hosts and service-principal identifiers for each environment. This is a deployment example, not a complete permissions or authentication configuration.

 targets:
  dev:
    default: true
    workspace:
      # Replace with the actual development workspace URL.
      host: https://adb-1234567890123456.7.azuredatabricks.net
    run_as:
      # Replace with the development workflow service principal identifier.
      service_principal_name: "00000000-0000-0000-0000-000000000000"
 
  test:
    workspace:
      # Replace with the actual test workspace URL.
      host: https://adb-3456789012345678.9.azuredatabricks.net
    run_as:
      # Replace with the test workflow service principal identifier.
      service_principal_name: "00000000-0000-0000-0000-000000000000"
 
  acc:
    mode: production
    workspace:
      # Replace with the actual acceptance workspace URL.
      host: https://adb-4567890123456789.0.azuredatabricks.net
      # Keep this reviewed path stable and distinct from other deployments.
      root_path: /Workspace/Applications/${bundle.name}/${bundle.target}
    run_as:
      # Replace with the acceptance workflow service principal identifier.
      service_principal_name: "00000000-0000-0000-0000-000000000000"
 
  prd:
    mode: production
    workspace:
      # Replace with the actual production workspace URL.
      host: https://adb-2345678901234567.8.azuredatabricks.net
      # Keep this reviewed path stable and distinct from other deployments.
      root_path: /Workspace/Applications/${bundle.name}/${bundle.target}
    run_as:
      # Replace with the production workflow service principal identifier.
      service_principal_name: "00000000-0000-0000-0000-000000000000"

The production run_as mapping makes the intended workflow identity explicit. Review deployment authentication and data-access permissions separately, and require execution tests under the intended workflow identity before promotion.

The CLI documentation identifies workspace.root_path as the basis of a bundle's unique deployment identity. Keep the production path stable, assign a deployment owner and prevent unrelated bundles from sharing it.

Keep credentials out of source files, YAML and pipeline variables. Prefer managed identities and workload identity federation. If secrets are necessary, use an environment-specific Azure Key Vault and a Key Vault-backed secret scope in Databricks. Follow the Azure Key Vault documentation and the relevant Databricks integration guidance.

4. Validate, deploy and test through CI

A few CLI commands do not constitute a production-ready CI pipeline. Run deployment commands only through shared CI/CD templates, with approval gates for acceptance and production.

set -euo pipefail
 
# Run inside the bundle folder and inspect the reported deployment identity.
databricks bundle validate
 
# Deploy explicitly to development only after validation succeeds.
databricks bundle deploy --target dev

Validation provides a configuration gate, while deployment creates or updates bundle-managed resources. Run the deployed workflow and check its outputs separately. Neither command proves that ingestion or transformation is correct.

Build the artefact once. Promote it through dev, test, acc and prd, using pull requests between dev, test, acc and main. Repeat target-specific checks before deployment. Retain the commit identifier, CLI version, validation output, deployment result and execution-test evidence for each release.

For a related example, see Deploying Databricks SAT with Declarative Automation Bundles Across DTAP. SAT's upgrade documentation describes bundle deployment as an advanced installation path; it is a useful deployment example, not proof that an arbitrary bundle is production-ready.

Pitfalls and When Not to Use It

Bundles fit projects with multiple contributors, repeatable deployment requirements and controlled promotion. For a disposable experiment with no promotion requirement, we would not introduce a full CI/CD framework just to deploy one notebook.

Do not treat mode: production as a security boundary. Deployment authentication, workflow identity, permissions and execution tests still need review, and target-specific behaviour must be tested rather than inferred from the mode name.

Resource deletion is part of deployment. The CLI documentation states that removing a previously deployed resource from bundle configuration removes it from the target workspace on deployment; review deletions as carefully as additions.

For an existing job or pipeline, do not assume that matching its display name adopts it into the bundle. The documented bundle deployment bind command associates an existing resource with a bundle definition, and the next deployment applies that definition. Inspect the existing resource and proposed changes first.

Do not routinely bypass deployment protections. The CLI documentation reserves --force-lock for a stale lock left by an interrupted deployment, and recommends --select only for development rather than production because downstream resources are not updated.

We recommend one deployment owner per resource. If Terraform and bundles coexist, document their boundaries instead of allowing both to manage the same job or pipeline; our Azure Terraform practices cover the surrounding infrastructure discipline.

Key Takeaways

1. Package release intent by reviewing application changes and resource configuration together.
2. Reuse shared templates while leaving transformation logic with product teams.
3. Prove workload behaviour with execution checks and expected data outputs.
4. Separate identity checks for deployment access and workflow execution.
5. Safeguard managed resources through stable deployment paths, deliberate binding and deletion reviews.
6. Preserve release evidence throughout promotion without overlapping Terraform and bundle ownership.