Inference on Azure Databricks: What Drives Cost and Latency

A document-extraction pilot can become an expensive service without anyone training a model. The spend comes from inference: running a trained model on new inputs to produce outputs. The architectural question is not simply which model to use, but how often to run it, how quickly it must respond and who pays for unused capacity.

We treat governance, deployment and operating cost as connected platform decisions. An endpoint needs an owner, an access boundary, a release process and evidence that its operating model fits the workload.

This post explains what inference does, why tokens and traffic shape matter, and how real-time and batch workloads differ. We then turn those mechanics into a practical serving decision for Azure Databricks.

The Short Version

Inference uses an existing model; it does not normally update that model's learned parameters. Real-time inference answers requests within an application deadline, while batch inference processes a dataset within a completion window. Token-based services charge for model usage, provisioned offerings allocate capacity, and self-hosting makes infrastructure utilisation and operations your responsibility. Start with the simplest governed option that meets your quality and latency requirements, then change the serving model when measurements justify it.

How It Works

From trained parameters to an output

Training adjusts a model's learned parameters using data and an optimisation objective. Fine-tuning is further training of an existing model for a particular task or behaviour. Inference applies those parameters to an input without performing that adjustment.

For a classification model, the input might be shipment attributes and the output a risk score. For a large language model, or LLM, the input is represented as tokens and the output is usually a generated token sequence. Saving that output in a Delta table does not make the model learn from it.

A token is a unit produced by a model's tokenizer, which converts text into model inputs. Tokens are not interchangeable with words or characters: different text and tokenizers produce different counts. Measure them using the model or provider's usage information rather than a character-count shortcut.

Real-time and batch are delivery choices

Real-time inference sits in a request-response path: an application submits input and waits for an answer. Batch inference processes a collection of inputs, usually asynchronously, with a deadline for the collection rather than each individual request.

The Azure Databricks Model Serving overview describes a unified interface for real-time and batch inference. It also documents integration with AI Functions, including ai_query, for inference inside data-processing workflows. With a pre-provisioned, Databricks-hosted model, that batch route does not require you to configure a serving endpoint yourself.

These choices are independent of model ownership. A scheduled job can call a hosted foundation model, while an interactive application can call a custom model.

Batch is not automatically cheaper. Calling the same token-priced service with the same inputs and outputs can produce similar inference charges, while also adding orchestration compute. Batch gives us scheduling freedom, controlled concurrency and opportunities to avoid repeated work — savings require those opportunities to be used.

What drives latency

LLM request time includes network transport, queueing, input processing and output generation. Prefill processes the input tokens; decoding generates the output tokens successively, using the input and previously generated output.

Long prompts increase input-processing work. Long responses occupy serving capacity for longer, and can dominate total response time even when the first token arrives quickly.

Concurrency is the number of requests in flight at the same time. Increasing concurrency can improve throughput until serving capacity becomes saturated; beyond that point, queueing or request rejection becomes the limiting factor.

For interactive applications, measure both time to first token, the delay before generation becomes visible, and completion latency. For extraction workflows, measure time to a validated result: an early token is not useful if the final response fails validation.

What drives cost

The relevant cost drivers depend on the charging model. Batch is a workflow choice, not a billing model — its costs combine inference with the compute needed to run the pipeline.

For token billing, estimate the charge as input token count multiplied by the input rate, plus output token count multiplied by the output rate. Use the applicable model, tier and billing rules, including any cached-input treatment — this is not a substitute for reconciling the invoice.

For capacity-based serving, utilisation matters. A predictable workload can keep capacity productive, whereas an endpoint sized for an occasional peak can spend much of its life underused.

GPU time is time for which accelerator capacity is allocated or billed, depending on the infrastructure contract. It is not the same as time spent doing useful inference. Small requests, inefficient batching and idle replicas can all consume paid capacity without producing much accepted output; many conventional ML models can instead run efficiently on CPUs.

What Azure Databricks manages

The Foundation Model APIs documentation describes pay-per-token, provisioned throughput and AI Functions optimised models for batch inference. It also documents priority pay-per-token for latency-sensitive requests: priority admission costs more per token and requires no commitment.

Databricks recommends provisioned throughput for production workloads, particularly those requiring high throughput, performance guarantees, supported fine-tuned models or additional security requirements. It also recommends priority pay-per-token for latency-sensitive production workloads needing more consistent performance under load. We treat these as vendor recommendations to test against workload evidence, not proof that every production workload needs provisioned capacity.

Custom models packaged in MLflow format can also run on managed Model Serving. Self-hosting means operating the serving infrastructure ourselves; uploading our own model to a managed serverless endpoint is not self-hosting.

External models provide a centrally managed Databricks access path to models hosted elsewhere. That governance boundary does not remove the external provider's processing, billing or residency considerations.

Hands-On

1. Define the contract before selecting capacity

Write down the acceptable output, completion deadline, expected traffic shape and permitted processing locations. An extraction service needs a validation contract, not just a prompt; an interactive assistant needs a latency budget for the whole application, not just model generation.

Use different evidence gates across development, test, acceptance and production. Each environment should answer a distinct question before we promote the workload.

Freeze the evaluation dataset and prompt version during comparisons. Otherwise, a change in input shape can look like a serving improvement.

2. Capture request shape and accepted results

Record usage for each attempt, including retries, and distinguish generated responses from accepted outputs. Keep sensitive prompts out of ordinary operational logs; token counts, timings, model identifiers and correlation IDs usually provide the first layer of evidence.

The following standard-library script analyses a caller-maintained JSON Lines trace. These field names are our logging contract, not Databricks API response fields.

# analyse_inference.py
# Each JSON line contains:
# input_tokens, output_tokens, elapsed_ms, accepted
# Record one line per attempt; accepted must be a JSON boolean.
import json
import math
import statistics
import sys

if len(sys.argv) != 2:
    raise SystemExit("Usage: python analyse_inference.py inference_attempts.jsonl")

with open(sys.argv[1], encoding="utf-8") as trace_file:
    attempts = [json.loads(line) for line in trace_file if line.strip()]

if not attempts:
    raise SystemExit("No inference attempts recorded")

if any(not isinstance(row["accepted"], bool) for row in attempts):
    raise SystemExit("The accepted field must be a JSON boolean")

latencies = sorted(float(row["elapsed_ms"]) for row in attempts)
accepted = sum(row["accepted"] for row in attempts)

# Nearest-rank percentile; use a sufficiently representative trace.
p95_index = math.ceil(0.95 * len(latencies)) - 1

summary = {
    "attempts": len(attempts),
    "accepted_outputs": accepted,
    "input_tokens": sum(int(row["input_tokens"]) for row in attempts),
    "output_tokens": sum(int(row["output_tokens"]) for row in attempts),
    "median_attempt_ms": statistics.median(latencies),
    "p95_attempt_ms": latencies[p95_index],
}
print(json.dumps(summary, indent=2))

Run python analyse_inference.py inference_attempts.jsonl against traces from each candidate configuration, using reported usage rather than estimated text length. Attempt latency helps diagnose serving behaviour, but user-visible latency must also include retry delays, retrieval and validation.

Compare total expenditure per accepted output, not merely per request. A cheaper model can be more expensive operationally if its failures trigger repeated calls or manual correction.

3. Reconcile operational traces with billing

Endpoint telemetry explains workload shape; billing records explain charged usage. Use both, and include job compute or external provider charges where applicable.

Databricks' model-serving cost-monitoring documentation provides the following pattern for examining serving usage. That page is AWS-scoped, so this is a starting point for validation rather than an established Azure-wide billing query.

TODO (editor): Verify the serving SKU filter and DBU coverage against Azure documentation and representative Azure billing records before using this query for cost allocation.

-- Inspect usage for the documented serving SKU pattern.
-- This is not a complete cost calculation or a filter for every offering.
SELECT
    usage_date,
    usage_metadata.endpoint_name AS endpoint_name,
    SUM(usage_quantity) AS model_serving_dbus
FROM system.billing.usage
WHERE sku_name LIKE '%SERVERLESS_REAL_TIME_INFERENCE%'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY
    usage_date,
    usage_metadata.endpoint_name
ORDER BY
    usage_date DESC,
    model_serving_dbus DESC;

This aggregates reported usage, not currency, and it does not establish coverage of every inference offering. Access requires the relevant system table to be enabled and appropriate permissions; reconcile the results with actual billing records and contracted rates before allocating costs.

An endpoint can also serve several applications. Endpoint-level billing alone cannot establish consumer-level allocation, a distinction we explore in our Databricks chargeback guide.

Why It Matters

Serving is a platform responsibility once an application depends on it. Model quality, capacity, identity, network access and deployment lifecycle all affect whether the application remains usable.

Our recommendation is to optimise the workflow before committing to infrastructure. Remove unnecessary context, bound the required response, avoid inference on unchanged data and validate outputs before downstream use. These changes reduce work regardless of the serving provider.

Do not buy latency consistency without understanding the application deadline. A document pipeline with a completion window can tolerate queueing that would make an interactive application unusable.

Governance can override the cheapest option. The Azure Model Serving documentation states that Azure Private Link support for respecting workspace networking-related ingress rules applies to provisioned-throughput endpoints and custom-model endpoints. Verify that the selected serving mode satisfies the required network boundary before comparing prices — this statement does not establish every private-connectivity path.

Use Case: Hybrid Logistics Document Processing

Consider a hypothetical logistics organisation whose operational systems remain partly on-premises while its analytics platform runs on Azure. Back-office teams need to classify documents and extract shipment references, but peak-season workloads arrive unevenly and operations staff cannot maintain another always-on infrastructure estate.

The usual pilot sends every document to an interactive endpoint and stores whatever comes back. That approach turns a back-office completion requirement into a low-latency serving requirement, repeats inference during pipeline reruns and leaves invalid outputs mixed with usable records.

We would ingest approved document text into governed Delta tables, select only new or changed records, and use a scheduled AI Functions pipeline for extraction. The pipeline would persist model and prompt versions alongside results, validate required fields and route ambiguous cases to human review. A separate interactive serving path would handle operator questions, keeping that application's latency requirements distinct from document throughput.

Measure completion against the operational deadline, expenditure per accepted document, repeated processing of unchanged inputs, validation failures and manual-review demand. For the interactive path, track completion latency and rejected requests under representative concurrent traffic. Those signals determine whether batch remains sufficient and whether interactive demand justifies priority or provisioned serving.

The output publication gate follows the same principle as our data-contract approach: successful execution is not proof that a result is fit for consumption. A structurally valid extraction can still contain incorrect shipment references.

Decision Guide

Choose the serving path by workload contract, then verify model support, region availability, limits and commercial terms. Compare full operating cost at the required quality level, not just the headline serving rate.

For an Azure-hosted provider alternative, Microsoft Foundry provisioned throughput uses provisioned throughput units, or PTUs, with capacity-based billing. Those are not interchangeable with Databricks capacity units. Microsoft also distinguishes quota permission from deployable capacity: having quota does not guarantee that capacity is available.

Our default is managed serving, batch wherever the deadline permits, and token billing for uncertain demand until workload evidence supports capacity allocation. Choose self-hosting because its control is necessary — not because GPU hourly pricing appears cheaper in isolation.

Pitfalls and Misconceptions

More requests do not tell the whole story

Requests per second is insufficient for LLM sizing. Input length, output length, concurrent requests and their distribution determine how much work the service must perform.

Test long and difficult inputs as well as typical ones. An average-only load test hides the cases most likely to breach the application deadline.

Batch and managed serving do not remove operational work

Persist validated results so retries do not automatically repeat successful inference. Where outputs feed governed datasets, include source identity and model and prompt versions in the processing key.

Managed infrastructure also does not remove dependency maintenance. The Model Serving documentation states that existing model images are not security-patched in place; newly created images from new model versions contain the latest patches. Plan rebuilds and regression tests rather than assuming a running image is continuously updated.

Platform location is not processing location

The Foundation Model APIs documentation describes residency through Databricks Geos. Do not interpret the workspace's region alone as proof that all model processing stays in that exact region.

Check the chosen model, service mode and processing boundary. Likewise, do not import AWS reserved-throughput settings, reservation terms or supported-model lists into an Azure deployment without Azure-specific confirmation.

Sometimes inference is not worth adopting

Use deterministic rules for exact mappings, simple validation and stable business policy. An LLM should not replace a reliable lookup or schema check merely because it can produce an answer.

Likewise, self-hosting is usually unjustified for intermittent traffic without a requirement that managed services cannot meet. The platform team's time belongs in the cost model.

Key Takeaways

1. Separate inference from training: running a model does not normally update its learned parameters.
2. Choose batch or real-time from the business deadline, not the pilot's implementation.
3. Measure token shape, concurrency, retries and accepted outputs before sizing capacity.
4. Start with managed token-based serving for uncertain demand; evaluate provisioned capacity using representative traffic.
5. Reconcile request telemetry with billing, pipeline compute and applicable provider charges.
6. Verify Azure-specific networking, residency, model support and capacity before promoting the serving choice to production.