Inference on Azure Databricks: What Drives Cost and Latency

A document-extraction pilot can become an expensive service without anyone training a model. The spend comes from inference: running a trained model on new inputs to produce outputs. The architectural questions concern model choice, request frequency, response deadlines and payment for unused capacity.

We treat governance, deployment and operating cost as connected platform decisions. An endpoint needs an owner, an access boundary, a release process and evidence that its operating model fits the workload.

Inference costs depend on workload shape; we recommend governed, measured serving, so define your deadline and compare cost per accepted output before allocating capacity.

The Short Version

Inference uses an existing model; it does not normally update that model's learned parameters. Real-time inference answers requests within an application deadline, while batch inference processes a dataset within a completion window. Token-based services charge for model usage, provisioned offerings allocate capacity, and self-hosting makes infrastructure utilisation and operations your responsibility. Start with the simplest governed option that meets your quality and latency requirements, then change the serving model when measurements justify it.

How It Works

From trained parameters to an output

Training adjusts a model's learned parameters using data and an optimisation objective. Fine-tuning is further training of an existing model for a particular task or behaviour. Inference applies those parameters to an input without performing that adjustment.

For a classification model, the input might be shipment attributes and the output a risk score. For a large language model, or LLM, the input is represented as tokens. The output is usually a generated token sequence. Saving that output in a Delta table does not make the model learn from it.

A token is a unit produced by a model's tokenizer, which converts text into model inputs. Tokens are not interchangeable with words or characters: different text and tokenizers produce different counts. Measure them using the model or provider's usage information rather than a character-count shortcut.

Real-time and batch are delivery choices

Real-time inference sits in a request-response path: an application submits input and waits for an answer. Batch inference processes a collection of inputs, usually asynchronously, with a deadline for the collection rather than each individual request.

The Azure Databricks Model Serving overview describes a unified interface for real-time and batch inference. It also documents integration with AI Functions, including ai_query, for inference inside data-processing workflows. With a pre-provisioned, Databricks-hosted model, that batch route does not require you to configure a serving endpoint yourself.

These choices are independent of model ownership. A scheduled job can call a hosted foundation model, while an interactive application can call a custom model.

What drives latency

LLM request time includes network transport, queueing, input processing and output generation. Prefill processes the input tokens; decoding generates the output tokens successively, using the input and previously generated output.

Long prompts increase input-processing work. Long responses occupy serving capacity for longer, and can dominate total response time even when the first token arrives quickly.

Concurrency is the number of requests in flight at the same time. Increasing concurrency can improve throughput until serving capacity becomes saturated; beyond that point, queueing or request rejection becomes the limiting factor.

For interactive applications, measure both time to first token and completion latency. Time to first token measures the delay before generation becomes visible. For extraction workflows, measure time to a validated result.

What drives cost

The relevant cost drivers depend on the charging model. Batch is a workflow choice, not a billing model. Its costs combine inference with the compute needed to run the pipeline.

For token billing, estimate input and output charges separately, then add them. Multiply each token count by its applicable rate. Use the model, tier and billing rules, including any cached-input treatment.

For capacity-based serving, utilisation matters. A predictable workload can keep capacity productive, whereas an endpoint sized for an occasional peak can spend much of its life underused.

GPU time is time for which accelerator capacity is allocated or billed, depending on the infrastructure contract. It is not the same as time spent doing useful inference. Small requests, inefficient batching and idle replicas can consume paid capacity without producing much accepted output. Many conventional ML models can instead run efficiently on CPUs.

What Azure Databricks manages

The Foundation Model APIs documentation describes pay-per-token, provisioned throughput and AI Functions optimised models for batch inference. It also documents priority pay-per-token for latency-sensitive requests: priority admission costs more per token and requires no commitment.

Databricks recommends provisioned throughput for production workloads, particularly those requiring high throughput, performance guarantees, supported fine-tuned models or additional security requirements. It also recommends priority pay-per-token for latency-sensitive production workloads needing more consistent performance under load. We test these vendor recommendations against workload evidence.

Custom models packaged in MLflow format can also run on managed Model Serving. Self-hosting means operating the serving infrastructure ourselves; uploading our own model to a managed serverless endpoint is not self-hosting.

External models provide a centrally managed Databricks access path to models hosted elsewhere. That governance boundary does not remove the external provider's processing, billing or residency considerations.

Hands-On

1. Define the contract before selecting capacity

Record the acceptable output, completion deadline, expected traffic shape and permitted processing locations. Define a validation contract for extraction. Set an application-wide latency budget for interactive assistants.

Use separate dev, test, acc and prd environments. Define a distinct evidence gate for each environment. Promote changes through pull requests from dev to test, then acc, then main for production.

Freeze the evaluation dataset and prompt version during comparisons. Otherwise, a change in input shape can look like a serving improvement.

Deploy infrastructure through CI/CD with version-pinned NexOps-managed Terraform modules. Deploy jobs, pipelines and serving resources through CI/CD with Databricks Asset Bundles. Use workload identity federation for deployment and service principals for jobs. Define Entra ID group access in code.

2. Capture request shape and accepted results

Record usage for each attempt, including retries. Distinguish generated responses from accepted outputs. Keep sensitive prompts out of ordinary operational logs. Capture token counts, timings, model identifiers and correlation IDs.

The following standard-library script analyses a caller-maintained JSON Lines trace. These field names are our logging contract, not Databricks API response fields.

# analyse_inference.py
# Each JSON line contains:
# input_tokens, output_tokens, elapsed_ms, accepted
# Record one line per attempt; accepted must be a JSON boolean.
import json
import math
import statistics
import sys
 
if len(sys.argv) != 2:
    raise SystemExit("Usage: python analyse_inference.py inference_attempts.jsonl")
 
with open(sys.argv[1], encoding="utf-8") as trace_file:
    attempts = [json.loads(line) for line in trace_file if line.strip()]
 
if not attempts:
    raise SystemExit("No inference attempts recorded")
 
if any(not isinstance(row["accepted"], bool) for row in attempts):
    raise SystemExit("The accepted field must be a JSON boolean")
 
latencies = sorted(float(row["elapsed_ms"]) for row in attempts)
accepted = sum(row["accepted"] for row in attempts)
 
# Nearest-rank percentile; use a sufficiently representative trace.
p95_index = math.ceil(0.95 * len(latencies)) - 1
 
summary = {
    "attempts": len(attempts),
    "accepted_outputs": accepted,
    "input_tokens": sum(int(row["input_tokens"]) for row in attempts),
    "output_tokens": sum(int(row["output_tokens"]) for row in attempts),
    "median_attempt_ms": statistics.median(latencies),
    "p95_attempt_ms": latencies[p95_index],
}
print(json.dumps(summary, indent=2))

Run python analyse_inference.py inference_attempts.jsonl against traces from each candidate configuration. Use reported usage rather than estimated text length. Measure user-visible latency separately, including retry delays, retrieval and validation. The script reports attempt latency.

Compare total expenditure per accepted output, not merely per request. A cheaper model can be more expensive operationally if its failures trigger repeated calls or manual correction.

3. Reconcile operational traces with billing

Endpoint telemetry explains workload shape; billing records explain charged usage. Use both, and include job compute or external provider charges where applicable.

Databricks' AWS-scoped model-serving cost-monitoring documentation documents a serving SKU pattern for querying DBU usage. The following query follows that pattern.

-- Inspect usage for the documented serving SKU pattern.
-- This is not a complete cost calculation or a filter for every offering.
SELECT
    usage_date,
    usage_metadata.endpoint_name AS endpoint_name,
    SUM(usage_quantity) AS model_serving_dbus
FROM system.billing.usage
WHERE sku_name LIKE '%SERVERLESS_REAL_TIME_INFERENCE%'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY
    usage_date,
    usage_metadata.endpoint_name
ORDER BY
    usage_date DESC,
    model_serving_dbus DESC;

The query groups reported DBU usage by day and endpoint. It requires access to the enabled billing usage system table.

Why It Matters

Serving is a platform responsibility once an application depends on it. Model quality, capacity, identity, network access and deployment lifecycle all affect whether the application remains usable.

Our recommendation is to optimise the workflow before committing to infrastructure. Reducing unnecessary context, response length and repeated inference reduces work regardless of the serving provider.

The business deadline determines the value of latency consistency. A document pipeline with a completion window can tolerate queueing that would make an interactive application unusable.

Use Case: Hybrid Logistics Document Processing

Consider a hypothetical logistics organisation whose operational systems remain partly on-premises while its analytics platform runs on Azure. Back-office teams need to classify documents and extract shipment references. Peak-season workloads arrive unevenly, and operations staff cannot maintain another always-on infrastructure estate.

The usual pilot sends every document to an interactive endpoint and stores whatever comes back. That approach turns a back-office completion requirement into a low-latency serving requirement. It also repeats inference during pipeline reruns and mixes invalid outputs with usable records.

We would ingest approved document text into governed Delta tables and select only new or changed records. A scheduled AI Functions pipeline would perform extraction through an approved private serving path. The pipeline would persist model and prompt versions alongside results, validate required fields and route ambiguous cases to human review. A separate interactive serving path would handle operator questions, keeping that application's latency requirements distinct from document throughput.

Measure deadline compliance, expenditure per accepted document, repeated processing, validation failures and manual-review demand. For the interactive path, track completion latency and rejected requests under representative concurrent traffic. Use those signals to assess batch suitability and interactive capacity needs.

The publication gate follows our data-contract approach: execution success does not prove fitness for consumption. Validate extracted shipment references against the available source evidence before publication.

Decision Guide

Choose the serving path by workload contract. Require private network paths and approved processing locations before comparing eligible options. For EU clients, require EU processing locations. Verify model support, region availability, limits and commercial terms.

For an Azure-hosted provider alternative, Microsoft Foundry provisioned throughput uses provisioned throughput units, or PTUs, with capacity-based billing. Microsoft distinguishes quota permission from deployable capacity. The service allocates capacity at deployment time.

Our default is managed serving through private network paths, with batch wherever the deadline permits. We favour usage-based billing for uncertain demand when an eligible option meets that boundary. Workload evidence must support capacity allocation.

Pitfalls and Misconceptions

Billing evidence has scope limits

The SQL example uses an AWS-documented SKU pattern, not verified Azure-wide billing coverage. Confirm the SKU filter, usage units and offering coverage against Azure billing records before cost allocation. Reconcile usage with contracted rates, pipeline compute and external provider charges.

An endpoint can serve several applications, so endpoint-level billing alone cannot establish consumer-level allocation. We explore this distinction in our Databricks chargeback guide.

Batch does not guarantee lower charges

Calling the same token-priced service with identical inputs and outputs can produce similar inference charges. Batch can also add orchestration compute. Use scheduling freedom, controlled concurrency and persisted results to reduce unnecessary work.

Network requirements can exclude a serving mode

We require private network paths and disabled public access. The Azure Model Serving documentation limits Azure Private Link support for workspace networking-related ingress rules. That support applies to provisioned-throughput endpoints and custom-model endpoints. This statement does not establish every private-connectivity path.

Verify the complete application-to-model path before selecting a serving mode. If no compliant private path exists, exclude that option. Apply the same check to batch calls and external providers.

More requests do not tell the whole story

Requests per second is insufficient for LLM sizing. Input length, output length, concurrent requests and their distribution determine how much work the service must perform.

Test long and difficult inputs as well as typical ones. An average-only load test hides the cases most likely to breach the application deadline.

Batch and managed serving do not remove operational work

Persist validated results so retries do not automatically repeat successful inference. Where outputs feed governed datasets, include source identity and model and prompt versions in the processing key.

Managed infrastructure also does not remove dependency maintenance. Databricks states that existing model images are not security-patched in place. Newly created images from new model versions contain the latest patches. Plan rebuilds and regression tests rather than assuming a running image is continuously updated.

Platform location is not processing location

The Foundation Model APIs documentation describes residency through Databricks Geos. Do not interpret the workspace's region alone as proof that all model processing stays in that exact region.

Check the chosen model, service mode and processing boundary. Likewise, do not import AWS reserved-throughput settings, reservation terms or supported-model lists into an Azure deployment without Azure-specific confirmation.

Sometimes inference is not worth adopting

Use deterministic rules for exact mappings, simple validation and stable business policy. An LLM should not replace a reliable lookup or schema check merely because it can produce an answer.

Likewise, self-hosting is usually unjustified for intermittent traffic without a requirement that managed services cannot meet. The platform team's time belongs in the cost model.

Key Takeaways

1. Establish quality and deadline gates before comparing serving options.
2. Benchmark representative request shapes and account for unsuccessful attempts.
3. Match capacity commitments to observed demand and accepted-output economics.
4. Validate charge allocation against billing evidence and the complete workflow cost.
5. Reject serving paths that fail Azure-specific network, residency or availability checks.