Open Source: Databricks Lakehouse Terraform Boilerplate: Beyond Generated Code

AI can generate a plausible Terraform module faster than we can explain its inputs. That makes a folder of workspace, networking and catalogue definitions a weak reason to adopt a boilerplate. The useful part is the engineering judgement around that code: what it owns, which assumptions it makes and how we prove those assumptions still hold.

That is the angle we take when evaluating a Databricks Lakehouse Terraform starting point. We want shared infrastructure decisions to be reviewable and reusable, without pretending that every Azure estate has the same network, identity model or governance requirements. A successful deployment is the beginning of validation, not evidence of production readiness.

Generated Terraform leaves ownership and validation unresolved; we recommend judging a boilerplate by explicit boundaries, controlled changes and repeatable acceptance tests.

The Problem

Generated Infrastructure as Code can look convincing while hiding the hardest decisions. An input called enable_private_endpoints says nothing about which traffic becomes private, whether browser authentication works or whether the deployment runner can still reach the workspace. A catalogue map does not explain who owns grants or how existing governance objects are handled.

The same problem applies to hand-written templates. Calling a workspace “production-ready” because Terraform finished successfully confuses resource creation with operational acceptance. We need tests for permitted access, denied access, connectivity, logging and recovery.

Why It Matters for Platform Teams

The scarce resource is not Terraform syntax; it is accountable ownership of shared infrastructure. A platform team needs to know who controls the network, who manages workspace configuration and who may change data access. We favour smaller ownership boundaries over one root configuration that can replace everything.

An open-source starting point earns its place when teams can inspect those boundaries and contribute tested improvements. We would expect architectural decisions, migration guidance, acceptance checks and a clear upgrade process alongside the modules. AI-generated changes should meet the same review standard as human-written changes.

Current Azure requirements also matter more than historical defaults. The VNet-injection documentation states that, after 31 March 2026, new VNets require explicit outbound connectivity. An older template that assumes default outbound access can therefore produce a workspace whose classic compute cannot reach required services.

We also need to distinguish provisioning from governance. Unity Catalog is automatically enabled for Azure Databricks workspaces created after 9 November 2023, according to the current overview. We should inspect the existing configuration before deciding to create or assign governance resources, rather than assume every workspace needs a new metastore.

Our Azure Terraform practices cover the wider platform discipline. The boilerplate should make those decisions visible, rather than replace them with unexplained defaults.

Use Case: Shared Infrastructure for an Airport Operator

Consider a hypothetical airport operator with autonomous product teams building baggage, stand-allocation and turnaround analytics on a shared Azure Databricks platform. Operational events and scheduled extracts land in Azure storage, pipelines prepare governed datasets, and operations teams consume reporting outputs. A central platform team owns infrastructure, while security and audit teams need continuous evidence that workspaces follow policy.

The usual approach breaks when each team copies a starter repository and modifies it independently. One changes network rules, another broadens catalogue grants, and a third introduces a second owner for an existing resource. Generating more Terraform does not solve the resulting disagreement about ownership or safe upgrades.

We would use the boilerplate as a shared contract rather than a disposable scaffold. The central team would publish reviewed module versions, document externally managed dependencies and require connectivity and access tests before promotion. Product teams would contribute changes through the shared repository instead of accumulating permanent forks.

The data flow gives us concrete acceptance criteria. An authorised pipeline identity must read its landing data and write its intended datasets; an unrelated team identity must be denied. The deployment runner and security assessment workspace must reach the APIs they need without bypassing the agreed network controls.

We would measure module-version divergence, manual changes outside Terraform, failed access tests and the time required to promote a reviewed platform change. We would also track unresolved security findings and exceptions with owners and review dates. These are proposed measures, not results attributed to a client.

Implementation

1. Establish ownership before writing module calls

Start with an inventory of existing Azure and Databricks resources. Record each resource's owner, existing Terraform management and authorised deployment identity. Import or reference existing resources deliberately. Do not create competing owners.

Use architecture decision records to explain decisions that generated code cannot justify. Document the selected workspace plan against actual feature requirements. Explain how VNet injection and secure cluster connectivity work together. Record deployment and storage identities, catalogue layout and private connectivity paths, with public access disabled.

Do not justify Premium solely by claiming it is required for Unity Catalog: the supplied overview does not establish that requirement. The supplied inbound Private Link guide, however, explicitly lists a Premium workspace among its prerequisites.

2. Define environment boundaries explicitly

A directory structure helps organise inputs, but does not create isolation or approval gates. The following layout is a recommended convention, not a verified listing of the NexOps repository.

environments/
├── dev/
│   ├── backend.tfvars
│   └── terraform.tfvars
├── test/
│   ├── backend.tfvars
│   └── terraform.tfvars
├── acc/
│   ├── backend.tfvars
│   └── terraform.tfvars
└── prd/
    ├── backend.tfvars
    └── terraform.tfvars

Each backend.tfvars must select separate remote state with locking for its environment. Give each environment its own resource group or subscription, workspace, catalogues, Key Vault and service principal. Configure approval gates separately in the deployment system.

Keep credentials out of these files. Define backend access, state locking and recovery procedures, and ensure deployment identities have only the access needed for their scope. Commit the provider lock file and review dependency changes rather than silently accepting new versions.

3. Test networking and governance as behaviours

For VNet injection, Azure Databricks requires two dedicated subnets: a container subnet and a host subnet, sometimes called private and public respectively. They cannot be shared across workspaces or used for unrelated resources. With secure cluster connectivity, both subnets use private IPs; the names do not imply public exposure.

Verify subnet delegation, network security group associations, required rules and explicit outbound connectivity against the current documentation. VNet injection applies to the classic compute plane; it does not establish serverless networking. We should not present it as an alternative to secure cluster connectivity.

For Private Link, specify the connectivity path before choosing module inputs. Inbound UI/API access, browser authentication, classic compute connectivity and serverless access to Azure resources are different concerns. Test DNS and connectivity from users, compute and the CI runner separately; a private endpoint resource alone proves none of these paths.

For Unity Catalog, inspect the current metastore, workspace configuration, storage access and principals first. The object model distinguishes catalogues, schemas and assets from metastore-level objects such as storage credentials and external locations. Review privileges against their target securables rather than treating one default_grants map as a complete access policy.

4. Review and apply the same Terraform plan

The following commands assume a reviewed root configuration, the environment files above, installed Terraform and valid provider and backend authentication. Run them in CI/CD with workload identity federation, not as a local deployment procedure. Use NexOps-managed modules for infrastructure and Entra ID groups for access grants.

set -euo pipefail
 
# Run in CI/CD from the reviewed Terraform root directory.
terraform fmt -check -recursive
terraform init -backend-config=environments/dev/backend.tfvars
terraform validate
 
# Save the proposed changes so apply does not generate a different plan.
terraform plan \
  -var-file=environments/dev/terraform.tfvars \
  -out=dev.tfplan
 
terraform show -no-color dev.tfplan
 
# Apply in the pipeline only after the saved plan passes review.
terraform apply dev.tfplan

The backend configuration selects state settings; the variable file supplies root-module inputs. Applying dev.tfplan uses the saved plan rather than recalculating changes from the variable file.

In CI, install the approved Terraform version and check out the intended commit. Initialise the backend in each execution environment. Transfer the saved plan through a protected artefact mechanism and apply it only after the required approval. Treat plans and state as sensitive because they can contain confidential values.

Promote changes through pull requests from dev to test, then acc, then main, which deploys to prd. Require deployment approval in acc and prd. If a plan deletes or replaces a stateful resource, stop and obtain explicit approval.

Keep application deployment ownership separate from infrastructure ownership. Deploy jobs, pipelines, dashboards, apps and model serving resources with Databricks Asset Bundles through CI/CD. Our declarative CI/CD post explains the bundle approach; avoid having Terraform and a bundle independently manage the same workload resource.

5. Add security assessment as a separate acceptance layer

The Security Analysis Tool (SAT) can inspect Databricks account and workspace configurations, but it is not a certification or formal security attestation. SAT version 0.5.0 and higher requires Unity Catalog, and the Azure installation guide requires at least one SAT setup in a workspace per Azure subscription. Its documented installation permissions include subscription Reader, account administration, workspace administration and membership of the metastore admin group.

The following preparation commands come from the official Azure Terraform installation guide. Use SAT as a separately reviewed dependency, not an assumed feature of the lakehouse boilerplate.

git clone https://github.com/databricks-industry-solutions/security-analysis-tool.git
cd security-analysis-tool/terraform/azure
 
# Review the selected revision before using its installation files.
git rev-parse HEAD
 
# Populate required values using this revision's variable declarations.
cp template.tfvars terraform.tfvars

These commands prepare the installation files; they do not deploy SAT. Keep credentials out of terraform.tfvars. Review the selected revision and adapt its installation to our pipeline authentication and deployment standards.

After deployment, run the SAT Initializer Notebook successfully before running the Driver Notebook. Check the assessment identity's required API connectivity without bypassing the agreed network controls. Turn findings into owned review actions before deciding which checks should block releases.

Pitfalls and When Not to Use It

In an established estate, blindly creating networking, identities or metastores may conflict with centrally managed resources. Use references, imports or a narrower module instead of replacing working infrastructure to match a template.

Do not make prd a magic switch. Private connectivity, monitoring and stricter grants need explicit configuration and tests, not an environment-name comparison.

Avoid promising a fixed deployment duration or publishing pipeline YAML as ready to use without testing its prerequisites. We would publish measured results only with the tested configuration and prerequisites.

SAT also has operational limits. Its repository states that it is provided for exploration, as-is, without formal Databricks SLA support. Review findings with security teams and stakeholders.

For a single experimental workspace, a shared module programme may create more maintenance than it removes. For a serverless-only design, do not copy classic-compute networking by habit. Adopt the smallest configuration that satisfies the architecture and evidence requirements.

Key Takeaways

1. Prioritise engineering decisions over the volume of generated Terraform.
2. Check public availability and licensing before treating a starting point as an open-source dependency.
3. Prove network behaviour across the required user, compute and deployment paths.
4. Map existing governance ownership before introducing new resources or grants.
5. Carry reviewed changes through every environment, with protected plans and controlled approval.
6. Evaluate SAT separately, including its administrative access and support limitations.