Building RIVM’s Self-Service Data Onboarding Platform

Self-service data onboarding should remove repetitive configuration work, not ownership or review. We want domain experts to describe their data while the platform controls how that description becomes a deployed pipeline.

At RIVM, the Dutch National Institute for Public Health and the Environment, we delivered a national public health data platform on Azure with Unity Catalog governance, LLM operations and a self-service data onboarding platform. That verified delivery is the foundation of this post; the implementation examples below illustrate the approach rather than reproduce RIVM’s production configuration.

The design principle is straightforward: put a usable interface in front of a controlled deployment contract. To reduce repetitive onboarding work, we recommend separating configuration, validation, authorisation and execution; build that separation into your deployment process.

The Problem

Manual onboarding mixes several different responsibilities in one engineering task. Someone must describe the source, identify its owner, agree its handling requirements, configure ingestion and arrange deployment.

A ticket can capture those decisions, but it is a poor executable specification. When engineers translate prose into configuration, missing ownership and ambiguous schedules tend to surface late, after implementation has already started.

We should automate the repeatable translation, not pretend every source is routine. A self-service path works best when the platform already has supported ingestion patterns and can route exceptions back to engineering.

Why It Matters for Platform Teams

The real product is a configuration contract, not a form builder. A visual interface makes that contract accessible, while server-side validation and deployment controls make it enforceable.

We recommend separating four questions: is the configuration structurally valid, is the source supported, is the requested access authorised, and does the deployed pipeline work? Passing one check does not answer the others.

This also gives platform teams a manageable maintenance boundary. We can improve shared ingestion templates and validation rules without asking every domain team to become experts in Terraform or Databricks deployment tooling.

Public-sector governance makes this separation particularly useful. Classification, ownership, legal basis and purpose belong in the onboarding conversation, but collecting those fields does not itself prove compliance or enforce access controls.

Use Case: Onboarding a Public Health Surveillance Source

Consider a hypothetical national public health organisation with sensitive datasets, strict sovereignty requirements and domain teams that need to onboard sources without waiting for routine engineering work. A surveillance team maintains an internal relational database and needs a governed dataset for recurring analysis.

The starting flow runs from the source database through a shared ingestion framework into governed tables for authorised analysts. The source owner understands the meaning of the fields, while the platform team owns connectivity, deployment identities and supported ingestion patterns.

The usual ticket-based approach breaks when those responsibilities are compressed into an informal request. Engineers can discover column types, but they cannot infer the approved purpose, decide who should read the data or establish whether a field contains sensitive information.

Applying the self-service approach means collecting those decisions through a visual interface and producing a versioned configuration. We validate the contract, resolve approved identities and secret references, review the proposed changes, then deploy through the platform’s controlled automation.

For a relational source, discovery should return metadata visible to the configured database identity, not claim to find every table. Columns and keys may require additional database-specific queries, and discovered types still need review before becoming an ingestion contract.

We would measure request-to-approved-deployment time, validation failures by field, engineering interventions and first-run success. We would also check whether analysts receive the intended access and whether unauthorised identities are denied; speed alone is not a useful success criterion.

Implementation

1. Define the contract before building the interface

Start with the smallest supported source pattern and make ownership and handling requirements explicit. The following YAML is an illustrative application contract, not a Databricks bundle or a claim about RIVM’s schema.

# Application configuration, not native Databricks bundle syntax.
information:
  domain: public_health
  source_name: surveillance_cases
  classification: restricted
  owner_group: surveillance-data-owners
  reader_group: surveillance-data-readers
 
meta:
  legal_basis: "Record the approved legal basis here"
  purpose: "Approved surveillance analysis"
 
source:
  kind: relational
  connection_ref: surveillance-database
  # Resolve this reference through an approved secret-management path.
  credential_ref: surveillance-reader-credential
  entities:
    - name: case_summary
      source_schema: reporting
      source_table: case_summary

The contract contains references rather than database credentials, and its governance text is deliberately not a legal assertion. We must define what each classification means and how group names resolve to recognised platform identities before accepting a submission.

Treat the visual form and YAML editor as two views of the same contract. Neither should be able to bypass server-side validation, and errors should point users back to the relevant field.

2. Validate structure and business rules separately

A schema validator checks shape and types; business rules check whether the request makes sense for the supported onboarding path. We should reject unexpected fields rather than silently discard a misspelt ownership or classification field.

The example below assumes a configuration has already been parsed safely into Python dictionaries. It demonstrates a small validation boundary, not a complete ingestion schema.

from collections.abc import Mapping
 
 
def validate_fields(value: object, required: set[str], path: str) -> None:
    if not isinstance(value, Mapping):
        raise ValueError(f"{path}: expected a mapping")
 
    missing = required - value.keys()
    unexpected = value.keys() - required
    if missing or unexpected:
        raise ValueError(
            f"{path}: missing={sorted(missing)}, "
            f"unexpected={sorted(unexpected)}"
        )
 
    for field in required:
        text = value[field]
        if not isinstance(text, str) or not text.strip():
            raise ValueError(f"{path}.{field}: expected non-empty text")
 
 
def validate_governance(config: object) -> None:
    if not isinstance(config, Mapping):
        raise ValueError("configuration: expected a mapping")
 
    # Validate each nested section explicitly.
    validate_fields(
        config.get("information"),
        {
            "domain",
            "source_name",
            "classification",
            "owner_group",
            "reader_group",
        },
        "information",
    )
    validate_fields(
        config.get("meta"),
        {"legal_basis", "purpose"},
        "meta",
    )

Each checked section rejects unknown keys and blank values, but this code does not validate source entities or enforce permissions. A production implementation needs tests for every nested model, cross-field constraints and configuration migrations.

Scheduling deserves its own validation boundary. Validate the final scheduler representation. Test time-zone and daylight-saving behaviour before activation.

3. Establish identity, secrets and network boundaries

Resolve requested owners and readers against identities recognised by the target platform before generating access changes. A directory group name in YAML is not proof that the group exists at the correct scope or that it has the prerequisites for effective access.

Prefer managed identities where the source supports them. Store required credential material in Azure Key Vault, with configuration containing only references. Review user input, logs, temporary files and generated artefacts for secret exposure.

Azure App Service supports Node.js, Python and custom-container hosting, making it a possible host for a configuration application. Its hosting capabilities do not establish that a particular deployment has private-only ingress, correctly routed egress or access to internal file shares.

Document and test each required connection separately: application ingress, source discovery, secret retrieval, image retrieval and deployment access. Use private network paths and disable public access. Verify DNS, routes and authentication for each connection.

4. Generate deployment artefacts and validate them

Keep infrastructure changes and workload deployment as distinct stages. Use version-pinned NexOps-managed Terraform modules for infrastructure and access configuration, with grants assigned to Entra ID groups. Declarative Automation Bundles, formerly Databricks Asset Bundles, package workload artefacts and resource definitions. Deploy through CI/CD with workload identity federation.

Generate concrete names and values before bundle validation unless the generator deliberately uses documented bundle substitution syntax. Strings such as ${origin} should not be presented as native bundle variables without an explicit generation step.

The official bundle tutorial documents schema generation, validation and deployment. Define bundle targets for dev, test, acc and prd. Run the following commands in CI/CD from the directory containing the generated databricks.yml.

# Generate the schema for the installed CLI version.
databricks bundle schema > bundle_config_schema.json
 
# Validate the generated bundle before deployment.
databricks bundle validate
 
# Deploy the configured dev target through CI/CD after review.
databricks bundle deploy -t dev

The tutorial requires Databricks CLI version 0.218.0 or above, configured authentication and workspace files enabled. We should pin and test our deployment toolchain rather than treat that minimum as a production version recommendation.

The generated job must include a verified compute selection and task configuration; validation is not a substitute for a successful test run. The supplied tutorial uses AWS examples, so we must not copy its instance types into an Azure deployment.

5. Gate changes and prove the first run

Promote the same artefact through dev, test, acc and prd, using reviewed pull requests between environment branches. Require approval before changes in acc and prd. Review the exact configuration revision and proposed changes, with clear ownership for both data decisions and platform exceptions.

After deployment, test source connectivity, metadata compatibility, output reconciliation and effective access. A created job is not evidence that ingestion succeeded, and a successful ingestion run is not evidence that permissions are correct.

Optional AI-assisted descriptions belong outside the critical deployment path. Send only approved metadata and require written client agreement before using external AI services. Preserve human-written descriptions and require review. Generated text cannot establish a field’s meaning or its sensitivity.

Pitfalls and When Not to Use It

Discovery is not authorisation. A connection test proves that an identity can reach a source; it does not establish that ingestion, retention or onward access is approved.

Metadata is not enforcement. Classification and sensitive-field flags need defined downstream controls, and requested groups need effective-access tests rather than assumptions based on their names.

Hosting is not a security boundary. We should not assume that an App Service container can mount an internal SMB share or use domain-integrated authentication without validating the actual implementation. Nor should we present Databricks Apps as a drop-in replacement for a Docker deployment without checking its deployment, runtime and networking requirements.

Self-service needs an exception path. Sources requiring bespoke authentication, unusual data types, uncertain legal handling or significant transformation logic should return to engineering review. Forcing them through a standard template hides complexity rather than removing it.

Automation must remain accountable. Avoid promising that no engineer is needed: routine requests may follow a paved path, but platform maintenance, failed deployments and security exceptions still need owners. We should retain deployment evidence and a recovery procedure for partially completed changes.

Key Takeaways

1. Define the onboarding boundary. Choose supported source patterns before designing the form.
2. Assign accountable groups. Connect ownership and access requests to recognised platform identities.
3. Reject incomplete requests. Check structure and business rules without treating acceptance as authorisation.
4. Review deployment evidence. Inspect generated artefacts and verify ingestion and access after deployment.
5. Track service quality. Assess delivery time together with failures, interventions and reconciliation results.
6. Distinguish examples from client evidence. Keep illustrative implementation details separate from RIVM’s verified delivery.