Open Source: Databricks Lakehouse Terraform Boilerplate — Beyond Generated Code

AI can generate a plausible Terraform module faster than we can explain its inputs. That makes a folder of workspace, networking and catalogue definitions a weak reason to adopt a boilerplate. The useful part is the engineering judgement around that code — what it owns, which assumptions it makes and how we prove those assumptions still hold.

That is the angle for our open-source Databricks Lakehouse Terraform starting point. We want shared infrastructure decisions to be reviewable and reusable, without pretending that every Azure estate has the same network, identity model or governance requirements. A successful deployment is the beginning of validation, not evidence of production readiness.

We should judge a boilerplate by the contract it gives platform teams: explicit ownership, controlled changes and repeatable acceptance tests. In this revision, we separate those recommendations from repository capabilities that the supplied sources do not establish. We do not claim deployment timings, client adoption of this repository or security controls that have not been inspected.

The Problem

Generated Infrastructure as Code can look convincing while hiding the hardest decisions. An input called enable_private_endpoints says nothing about which traffic becomes private, whether browser authentication works or whether the deployment runner can still reach the workspace. A catalogue map does not explain who owns grants or how existing governance objects are handled.

The same problem applies to hand-written templates. Calling a workspace “production-ready” because Terraform finished successfully confuses resource creation with operational acceptance. We need tests for permitted access, denied access, connectivity, logging and recovery.

The original module examples depended on local module interfaces that are not included in the supplied source material. We therefore cannot present those inputs, outputs or conditional behaviours as verified implementation details.

TODO (editor): Verify the availability, licence, module interfaces and documented capabilities of https://github.com/nexops-nl/databricks-lakehouse-terraform before publishing it as a downloadable NexOps boilerplate.

Why It Matters for Platform Teams

The scarce resource is not Terraform syntax — it is accountable ownership of shared infrastructure. A platform team needs to know who controls the network, who manages workspace configuration and who may change data access. We favour smaller ownership boundaries over one root configuration that can replace everything.

An open-source starting point earns its place when teams can inspect those boundaries and contribute tested improvements. We would expect architectural decisions, migration guidance, acceptance checks and a clear upgrade process alongside the modules. AI-generated changes should meet the same review standard as human-written changes.

Current Azure requirements also matter more than historical defaults. The VNet-injection documentation states that, after 31 March 2026, new VNets require explicit outbound connectivity. An older template that assumes default outbound access can therefore produce a workspace whose classic compute cannot reach required services.

We also need to distinguish provisioning from governance. Unity Catalog is automatically enabled for Azure Databricks workspaces created after 9 November 2023, according to the current overview. We should inspect the existing configuration before deciding to create or assign governance resources — not assume every workspace needs a new metastore.

Our Azure Terraform practices cover the wider platform discipline. The boilerplate should make those decisions visible, rather than replace them with unexplained defaults.

Use Case: Shared Infrastructure for an Airport Operator

Consider a hypothetical airport operator with autonomous product teams building baggage, stand-allocation and turnaround analytics on a shared Azure Databricks platform. Operational events and scheduled extracts land in Azure storage, pipelines prepare governed datasets, and operations teams consume reporting outputs. A central platform team owns infrastructure, while security and audit teams need continuous evidence that workspaces follow policy.

The usual approach breaks when each team copies a starter repository and modifies it independently. One changes network rules, another broadens catalogue grants, and a third introduces a second owner for an existing resource. Generating more Terraform does not solve the resulting disagreement about ownership or safe upgrades.

We would use the boilerplate as a shared contract rather than a disposable scaffold. The central team would publish reviewed module versions, document externally managed dependencies and require connectivity and access tests before promotion. Product teams would contribute changes through the shared repository instead of accumulating permanent forks.

The data flow gives us concrete acceptance criteria. An authorised pipeline identity must read its landing data and write its intended datasets; an unrelated team identity must be denied. The deployment runner and security assessment workspace must reach the APIs they need without bypassing the agreed network controls.

We would measure module-version divergence, manual changes outside Terraform, failed access tests and the time required to promote a reviewed platform change. We would also track unresolved security findings and exceptions with owners and review dates. These are proposed measures, not results attributed to a client.

Implementation

1. Establish ownership before writing module calls

Start with an inventory of existing Azure and Databricks resources. Record which team owns each resource, whether Terraform already manages it and which deployment identity is allowed to change it. Import or reference existing resources deliberately — do not create competing owners.

Use architecture decision records to explain decisions that generated code cannot justify. Document the selected workspace plan against actual feature requirements, how VNet injection and secure cluster connectivity work together, and which identities authenticate deployments and access storage. Treat catalogue layout and private connectivity as design choices, not universal defaults.

Do not justify Premium solely by claiming it is required for Unity Catalog: the supplied overview does not establish that requirement. The supplied inbound Private Link guide, however, explicitly lists a Premium workspace among its prerequisites.

2. Define environment boundaries explicitly

A directory structure helps organise inputs, but does not create isolation or approval gates. The following layout is a recommended convention, not a verified listing of the NexOps repository.

environments/
├── dev/
│   ├── backend.tfvars
│   └── terraform.tfvars
├── test/
│   ├── backend.tfvars
│   └── terraform.tfvars
└── production/
    ├── backend.tfvars
    └── terraform.tfvars

Each backend.tfvars must resolve to the intended state location, with distinct state keys where environments need isolation. Approval gates belong in the deployment system and must be configured separately.

Keep credentials out of these files. Define backend access, state locking and recovery procedures, and ensure deployment identities have only the access needed for their scope. Commit the provider lock file and review dependency changes rather than silently accepting new versions.

3. Test networking and governance as behaviours

For VNet injection, Azure Databricks requires two dedicated subnets: a container subnet and a host subnet, sometimes called private and public respectively. They cannot be shared across workspaces or used for unrelated resources. With secure cluster connectivity, both subnets use private IPs — the names do not imply public exposure.

Verify subnet delegation, network security group associations, required rules and explicit outbound connectivity against the current documentation. VNet injection applies to the classic compute plane; it does not establish serverless networking. We should not present it as an alternative to secure cluster connectivity.

For Private Link, specify the connectivity path before choosing module inputs. Inbound UI/API access, browser authentication, classic compute connectivity and serverless access to Azure resources are different concerns. Test DNS and connectivity from users, compute and the CI runner separately; a private endpoint resource alone proves none of these paths.

For Unity Catalog, inspect the current metastore, workspace configuration, storage access and principals first. The object model distinguishes catalogues, schemas and assets from metastore-level objects such as storage credentials and external locations. Review privileges against their target securables rather than treating one default_grants map as a complete access policy.

4. Review and apply the same Terraform plan

The following commands assume a reviewed root configuration, the environment files above, installed Terraform and valid provider and backend authentication. They are a local workflow example — not evidence that the unverified repository deploys a complete platform.

set -euo pipefail

# Run from the reviewed Terraform root directory.
terraform fmt -check -recursive
terraform init -backend-config=environments/dev/backend.tfvars
terraform validate

# Save the proposed changes so apply does not generate a different plan.
terraform plan \
  -var-file=environments/dev/terraform.tfvars \
  -out=dev.tfplan

terraform show -no-color dev.tfplan

# Run only after reviewing the saved plan.
terraform apply dev.tfplan

The backend configuration selects state settings; the variable file supplies root-module inputs. Applying dev.tfplan uses the saved plan rather than recalculating changes from the variable file.

In CI, install the approved Terraform version, check out the intended commit, authenticate and initialise the backend in each execution environment. Transfer the saved plan through a protected artefact mechanism and apply it only after the required approval. Treat plans and state as sensitive because they can contain confidential values.

Keep application deployment ownership separate from infrastructure ownership. Our declarative CI/CD post explains the bundle approach; avoid having Terraform and a bundle independently manage the same workload resource.

5. Add security assessment as a separate acceptance layer

The Security Analysis Tool (SAT) can inspect Databricks account and workspace configurations, but it is not a certification or formal security attestation. SAT version 0.5.0 and higher requires Unity Catalog, and the Azure installation guide requires at least one SAT setup in a workspace per Azure subscription. Its documented installation permissions include subscription Reader, account administration, workspace administration and membership of the metastore admin group.

The following preparation commands come from the official Azure Terraform installation guide. Use SAT as a separately reviewed dependency, not an assumed feature of the lakehouse boilerplate.

git clone https://github.com/databricks-industry-solutions/security-analysis-tool.git
cd security-analysis-tool/terraform/azure

# Review the selected revision before using its installation files.
git rev-parse HEAD

# Populate required values using this revision's variable declarations.
cp template.tfvars terraform.tfvars

# Interactive login follows the documented local Azure installation path.
az login

Select and review a release or commit before deployment, then follow its installation instructions. Interactive az login is a local setup step, not a CI authentication design.

After deployment, run the SAT Initializer Notebook successfully before running the Driver Notebook. Check that the assessment identity can reach all intended workspaces: IP access lists can block analysis, and serverless egress controls can require a separate SAT deployment in a target workspace. Turn findings into owned review actions before deciding which checks should block releases.

Pitfalls and When Not to Use It

Do not adopt a boilerplate that cannot explain its ownership model. In an established estate, blindly creating networking, identities or metastores may conflict with centrally managed resources. Use references, imports or a narrower module instead of replacing working infrastructure to match a template.

Do not make production a magic switch. Private connectivity, monitoring and stricter grants need explicit configuration and tests, not an environment-name comparison. Likewise, forwarding logs is only part of monitoring: Log Analytics access depends on its access-control mode and Azure permissions.

Avoid promising a fixed deployment duration or publishing pipeline YAML as ready to use without testing installation, authentication, checkout, backend initialisation and plan transfer. Resource creation can succeed while DNS, data access or application deployment remains broken. We would publish measured results only with the tested configuration and prerequisites.

SAT also has operational limits. Its repository states that it is provided for exploration, as-is, without formal Databricks SLA support. Review findings with security teams and stakeholders, and account for dashboard access: embedded credentials can expose results to viewers who would not otherwise have access.

For a single experimental workspace, a shared module programme may create more maintenance than it removes. For a serverless-only design, do not copy classic-compute networking by habit. Adopt the smallest configuration that satisfies the architecture and evidence requirements.

Key Takeaways

1. Code generation is not the differentiator. A useful boilerplate supplies reviewed decisions, ownership boundaries, acceptance tests and upgrade guidance.
2. Verify repository capabilities. Do not infer working modules, security controls, licensing or production readiness from example inputs.
3. Test the actual connectivity paths. VNet injection, secure cluster connectivity and Private Link address different concerns.
4. Inspect existing governance first. Automatically enabled Unity Catalog does not remove the need to review storage access, principals and grants.
5. Promote reviewed plans and evidence. Separate state deliberately, protect plan artefacts and test authorised and denied behaviour.
6. Treat SAT as supporting evidence. Its permissions, network reachability and non-SLA support status need their own operational decisions.