Building RIVM’s Self-Service Data Onboarding Platform

Self-service data onboarding should remove repetitive configuration work—not ownership or review. We want domain experts to describe their data while the platform controls how that description becomes a deployed pipeline.

At RIVM, the Dutch National Institute for Public Health and the Environment, we delivered a national public health data platform on Azure with Unity Catalog governance, LLM operations and a self-service data onboarding platform. That verified delivery is the foundation of this post; the implementation examples below illustrate the approach rather than reproduce RIVM’s production configuration.

The design principle is straightforward: put a usable interface in front of a controlled deployment contract. We separate configuration, validation, authorisation and execution so that a successful form submission is never mistaken for permission to ingest sensitive data.

The Problem

Manual onboarding mixes several different responsibilities in one engineering task. Someone must describe the source, identify its owner, agree its handling requirements, configure ingestion and arrange deployment.

A ticket can capture those decisions, but it is a poor executable specification. When engineers translate prose into configuration, missing ownership and ambiguous schedules tend to surface late—after implementation has already started.

We should automate the repeatable translation, not pretend every source is routine. A self-service path works best when the platform already has supported ingestion patterns and can route exceptions back to engineering.

Why It Matters for Platform Teams

The real product is a configuration contract, not a form builder. A visual interface makes that contract accessible, while server-side validation and deployment controls make it enforceable.

We recommend separating four questions: is the configuration structurally valid, is the source supported, is the requested access authorised, and does the deployed pipeline work? Passing one check does not answer the others.

This also gives platform teams a manageable maintenance boundary. We can improve shared ingestion templates and validation rules without asking every domain team to become experts in Terraform or Databricks deployment tooling.

Public-sector governance makes this separation particularly useful. Classification, ownership, legal basis and purpose belong in the onboarding conversation, but collecting those fields does not itself prove compliance or enforce access controls.

Use Case: Onboarding a Public Health Surveillance Source

Consider a hypothetical national public health organisation with sensitive datasets, strict sovereignty requirements and domain teams that need to onboard sources without waiting for routine engineering work. A surveillance team maintains an internal relational database and needs a governed dataset for recurring analysis.

The starting flow is source database → shared ingestion framework → governed tables → authorised analysts. The source owner understands the meaning of the fields, while the platform team owns connectivity, deployment identities and supported ingestion patterns.

The usual ticket-based approach breaks when those responsibilities are compressed into an informal request. Engineers can discover column types, but they cannot infer the approved purpose, decide who should read the data or establish whether a field contains sensitive information.

Applying the self-service approach means collecting those decisions through a visual interface and producing a versioned configuration. We validate the contract, resolve approved identities and secret references, review the proposed changes, then deploy through the platform’s controlled automation.

For a relational source, discovery should return metadata visible to the configured database identity—not claim to find every table. Columns and keys may require additional database-specific queries, and discovered types still need review before becoming an ingestion contract.

We would measure request-to-approved-deployment time, validation failures by field, engineering interventions and first-run success. We would also check whether analysts receive the intended access and whether unauthorised identities are denied; speed alone is not a useful success criterion.

Implementation

1. Define the contract before building the interface

Start with the smallest supported source pattern and make ownership and handling requirements explicit. The following YAML is an illustrative application contract, not a Databricks bundle or a claim about RIVM’s schema.

# Application configuration—not native Databricks bundle syntax.
information:
  domain: public_health
  source_name: surveillance_cases
  classification: restricted
  owner_group: surveillance-data-owners
  reader_group: surveillance-data-readers

meta:
  legal_basis: "Record the approved legal basis here"
  purpose: "Approved surveillance analysis"

source:
  kind: relational
  connection_ref: surveillance-database
  # Resolve this reference through an approved secret-management path.
  credential_ref: surveillance-reader-credential
  entities:
    - name: case_summary
      source_schema: reporting
      source_table: case_summary

The contract contains references rather than database credentials, and its governance text is deliberately not a legal assertion. We must define what each classification means and how group names resolve to recognised platform identities before accepting a submission.

Treat the visual form and YAML editor as two views of the same contract. Neither should be able to bypass server-side validation, and errors should point users back to the relevant field.

2. Validate structure and business rules separately

A schema validator checks shape and types; business rules check whether the request makes sense for the supported onboarding path. We should reject unexpected fields rather than silently discard a misspelt ownership or classification field.

The example below assumes a configuration has already been parsed safely into Python dictionaries. It demonstrates a small validation boundary, not a complete ingestion schema.

from collections.abc import Mapping


def validate_fields(value, required, path):
    if not isinstance(value, Mapping):
        raise ValueError(f"{path}: expected a mapping")

    missing = required - value.keys()
    unexpected = value.keys() - required
    if missing or unexpected:
        raise ValueError(
            f"{path}: missing={sorted(missing)}, "
            f"unexpected={sorted(unexpected)}"
        )

    for field in required:
        text = value[field]
        if not isinstance(text, str) or not text.strip():
            raise ValueError(f"{path}.{field}: expected non-empty text")


def validate_governance(config):
    if not isinstance(config, Mapping):
        raise ValueError("configuration: expected a mapping")

    # Validate each nested section explicitly.
    validate_fields(
        config.get("information"),
        {
            "domain",
            "source_name",
            "classification",
            "owner_group",
            "reader_group",
        },
        "information",
    )
    validate_fields(
        config.get("meta"),
        {"legal_basis", "purpose"},
        "meta",
    )

Each checked section rejects unknown keys and blank values, but this code does not validate source entities or enforce permissions. A production implementation needs tests for every nested model, cross-field constraints and configuration migrations.

Scheduling deserves its own validation boundary. Do not pass a five-field Unix cron expression directly into a Quartz schedule—validate the final scheduler representation and test time-zone and daylight-saving behaviour before activation.

3. Establish identity, secrets and network boundaries

Resolve requested owners and readers against identities recognised by the target platform before generating access changes. A directory group name in YAML is not proof that the group exists at the correct scope or that it has the prerequisites for effective access.

We recommend an approved secret store such as Azure Key Vault for credential material, with configuration containing only references. Review user input, logs, temporary files and generated artefacts as well; using a vault does not prove that the application never handles or persists secrets.

Azure App Service supports Node.js, Python and custom-container hosting, making it a possible host for a configuration application. Its hosting capabilities do not establish that a particular deployment has private-only ingress, correctly routed egress or access to internal file shares.

Document and test each required connection separately: application ingress, source discovery, secret retrieval, image retrieval and deployment access. We should verify DNS, routes, public-access settings and authentication rather than describe everything as being inside one perimeter.

4. Generate deployment artefacts and validate them

Keep infrastructure changes and workload deployment as distinct stages. Terraform can manage the platform’s approved infrastructure and access configuration, while Databricks Asset Bundles package workload artefacts and resource definitions.

Generate concrete names and values before bundle validation unless the generator deliberately uses documented bundle substitution syntax. Strings such as ${origin} should not be presented as native bundle variables without an explicit generation step.

The official bundle tutorial documents schema generation, validation and deployment. From the directory containing the generated databricks.yml, we can use:

# Generate the schema for the installed CLI version.
databricks bundle schema > bundle_config_schema.json

# Validate the generated development target before deployment.
databricks bundle validate -t development

# Run only after the required review and approval.
databricks bundle deploy -t development

The tutorial requires Databricks CLI version 0.218.0 or above, configured authentication and workspace files enabled. We should pin and test our deployment toolchain rather than treat that minimum as a production version recommendation.

The generated job must include a verified compute selection and task configuration; validation is not a substitute for a successful test run. The supplied tutorial uses AWS examples, so we must not copy its instance types into an Azure deployment.

5. Gate changes and prove the first run

Place approval before the stages that change infrastructure or deploy production workloads. Review should cover the exact configuration revision and proposed changes, with clear ownership for both data decisions and platform exceptions.

After deployment, test source connectivity, metadata compatibility, output reconciliation and effective access. A created job is not evidence that ingestion succeeded, and a successful ingestion run is not evidence that permissions are correct.

Optional AI-assisted descriptions belong outside the critical deployment path. We recommend sending only approved metadata, preserving human-written descriptions and requiring review—generated text cannot establish a field’s meaning or its sensitivity.

Pitfalls and When Not to Use It

Discovery is not authorisation. A connection test proves that an identity can reach a source; it does not establish that ingestion, retention or onward access is approved.

Metadata is not enforcement. Classification and sensitive-field flags need defined downstream controls, and requested groups need effective-access tests rather than assumptions based on their names.

Hosting is not a security boundary. We should not assume that an App Service container can mount an internal SMB share or use domain-integrated authentication without validating the actual implementation. Nor should we present Databricks Apps as a drop-in replacement for a Docker deployment without checking its deployment, runtime and networking requirements.

Self-service needs an exception path. Sources requiring bespoke authentication, unusual data types, uncertain legal handling or significant transformation logic should return to engineering review. Forcing them through a standard template hides complexity rather than removing it.

Automation must remain accountable. Avoid promising that no engineer is needed: routine requests may follow a paved path, but platform maintenance, failed deployments and security exceptions still need owners. We should retain deployment evidence and a recovery procedure for partially completed changes.

Key Takeaways

1. Build around a contract. The visual interface should produce versioned configuration that the platform can validate and deploy.
2. Keep governance decisions explicit. Ownership, classification, legal basis and purpose need review and corresponding enforcement.
3. Separate validation from authorisation. Structurally correct YAML is not permission to ingest data or grant access.
4. Validate generated deployment artefacts. Check the final bundle, compute selection, schedule and identities—not just the application’s input.
5. Measure operational outcomes. Track onboarding time alongside first-run success, exceptions, reconciliation and effective access.
6. Keep the delivery claims precise. RIVM’s verified self-service platform delivery supports the case for this approach; illustrative architecture and code are not evidence of its production configuration.