Model Routing Is a Quality-Control Problem, Not a Cost Switch

A cheaper model call does not necessarily produce a cheaper completed task. Retries, failed validation and reviewer corrections can consume the apparent saving.

Model routing selects a model for a task according to a tested policy. That policy changes the execution path, so we treat it as a deployment decision rather than a pricing adjustment.

Before changing production traffic, we need evidence from complete tasks, not isolated responses. When cheaper models can increase task failures, we recommend treating routing as quality control: benchmark complete tasks before changing production traffic.

The Short Version

Routing needs evidence that the selected model can complete the assigned work. Use traces to classify real tasks, then benchmark candidates against common acceptance criteria. Keep task-aware selection in application code unless a verified platform capability meets the requirement. Release the policy only when quality, end-to-end latency and cost per successful outcome pass agreed gates.

How It Works

Those gates require clear boundaries between the routing policy, the agent application and the platform services.

What the sources establish

LangChain’s routing video provides the topic, but the supplied material does not establish reproducible benchmark results. We cannot infer workload quality, savings or statistical equivalence from it.

The Open SWE repository documents model customisation and engineering workflows. Its supplied README does not establish the video’s classifier, routing granularity or fallback behaviour.

Deploying Open SWE unchanged requires Managed Deep Agents. Without it, the repository recommends forking and adapting the code to another runtime.

Start with execution evidence

A trace records execution steps, including model calls, tool calls and their relationships. An agent harness is the application code that controls model calls, tools, state and validation.

Traces can reveal recurring task categories and failure paths. For coding work, useful categories include repository investigation, documentation changes, implementation and failure diagnosis.

A model tier groups candidates with similar measured cost and capability for your workload. Labels such as “economy” and “performance” are policy names, not guarantees.

A candidate belongs to the Pareto frontier when no alternative improves one selected measure without worsening another. Apply mandatory quality gates before comparing that frontier.

Separate application policy from platform routing

The sources support different parts of the architecture. They do not establish a turnkey routing integration across Open SWE, Azure and Databricks.

The external-model documentation describes percentage-based traffic routing when configuring multiple external models. This distributes requests; it does not establish semantic task classification or quality-driven escalation.

The supplied Databricks pages use the AWS documentation path. Azure availability, regional support and private connectivity need separate verification for the exact deployment.

External-model endpoints and Unity Gateway services also have different configuration surfaces. The integration design must identify the service, object type and API it uses.

Hands-On

With those boundaries defined, we can build a small routing experiment without inventing a platform integration.

1. Build a governed task dataset

1. Define a thread as one submitted task and all attempts through completion or termination.
2. Define success before running the experiment.
3. Create a versioned dataset from representative traces.
4. Include ordinary tasks, difficult tasks, failures and unknown categories.
5. Separate development examples from the held-out release set.

For coding work, acceptance can require focused tests, reviewer acceptance and no disallowed changes. A merged pull request alone does not prove correctness.

Classify data before ingestion. Remove secrets and unnecessary personal data before storing traces. Record the processing purpose, data owner and retention period.

Protect sensitive fields and restrict access through Entra ID groups. Use synthetic, masked or anonymised inputs in development and test. Keep EU client data in agreed EU regions.

Obtain written client approval before sending client data to external AI services. Apply this requirement to evaluation and shadow execution as well as production.

2. Benchmark complete tasks

1. Run each candidate against the same fixed evaluation set.
2. Keep prompts, tool permissions, repository revisions and validation rules consistent.
3. Evaluate results by task category and risk level.
4. Record quality, tail latency, retries, tool failures and total task cost.
5. Include classifier calls and unsuccessful attempts.
6. Calibrate model-assisted grading against human-reviewed examples.

Start with deterministic, thread-level selection. Treat per-step selection, subagent routing and mid-thread escalation as separate experiments.

Define the experimental unit before allocating traffic. Keep related work together when shared context could contaminate the comparison.

3. Implement an explicit policy

Use a conservative policy for the first experiment. Route only approved low-risk categories to the economy tier.

The following is custom application code, not an Open SWE extension API or a Databricks configuration. It selects a logical route without creating endpoints or calling models.

# src/routing.py
from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class Task:
    category: str
    risk: str
 
 
def select_route(
    task: Task,
    *,
    eligible_routes: frozenset[str],
    approved_economy_tasks: frozenset[str] = frozenset(),
) -> str:
    """Select an authorised route without calling a model."""
    selected = "performance"
 
    # Unapproved categories and risk levels retain the baseline.
    if task.risk == "low" and task.category in approved_economy_tasks:
        selected = "economy"
 
    # Model selection must not bypass destination authorisation.
    if selected not in eligible_routes:
        raise PermissionError("Selected route is not authorised")
 
    return selected

The empty allowlist keeps all tasks on the baseline route. The function rejects an unauthorised selection rather than silently choosing another provider.

1. Compute eligible_routes from destination permissions, regional restrictions and verified private network paths.
2. Derive risk from trusted policy and task metadata.
3. Unit-test unknown categories, unknown risk values and unauthorised destinations.
4. Record the policy version and selection reason for every thread.
5. Keep tool permissions and approval requirements independent of model selection.

4. Correlate telemetry with outcomes

MLflow Tracing provides execution evidence for debugging and evaluation. It supports standard MLflow and OpenTelemetry instrumentation, with custom tracing where automatic instrumentation does not cover the application.

For a supported LangChain application, the documented setup includes the following calls:

# src/observability.py
import os
 
import mlflow
 
mlflow.set_tracking_uri("databricks")
mlflow.set_experiment(os.environ["MLFLOW_EXPERIMENT_NAME"])
 
# Explicitly enable tracing for supported LangChain calls.
mlflow.langchain.autolog()

The experiment name comes from environment-specific configuration, not a secret. Run this setup before invoking the application’s supported LangChain components.

Pin compatible dependencies in the lock file. Authenticate through the approved workload identity configuration. Verify captured spans with synthetic tasks before using the integration for evaluation.

Unity Gateway usage tracking records usage in system.ai_gateway.usage. The supplied schema documents request identifiers, invocation identifiers, latency and request tags, but does not show the token columns.

The following query therefore summarises records and latency without assuming a total_tokens column:

SELECT
    request_tags['experiment_arm'] AS experiment_arm,
    endpoint_name,
    destination_model,
    COUNT(*) AS usage_records,
    AVG(latency_ms) AS mean_record_latency_ms
FROM system.ai_gateway.usage
WHERE
    event_time >= CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
    -- Exclude MCP records from this model-call summary.
    AND service_type IN (
        'MODEL_SERVICE',
        'MODEL_PROVIDER_SERVICE'
    )
GROUP BY
    request_tags['experiment_arm'],
    endpoint_name,
    destination_model;

experiment_arm is our tag convention, not a built-in field. The query counts usage records, not completed tasks or necessarily distinct model calls.

1. Verify request-tag propagation through the client and gateway.
2. Record the relationship between thread IDs, trace IDs and gateway request identifiers.
3. Check record semantics and duplicate accounting across service types.
4. Inspect the deployed schema before adding token aggregation.
5. Reconcile telemetry with the outcome dataset.

Usage tracking is billed and requires Unity Catalog and a supported region. Define narrow, group-based read grants in Terraform.

Store outcome datasets in Unity Catalog with group ownership. Use bronze, silver and gold layers for raw telemetry, reconciled records and evaluation summaries.

5. Define release gates and promote through DTAP

Use cost per successful outcome as a primary measure. Calculate it as total experiment-arm cost divided by successful tasks in that arm.

Include failed tasks in the numerator. If there are no successes, report the metric as undefined.

Include model, gateway and sandbox costs within a documented accounting boundary. Report reviewer effort separately unless an agreed method converts it into cost.

Databricks Model Serving cost documentation describes billing evidence in system.billing.usage. This evidence is separate from task outcomes and does not automatically establish complete external-provider spend.

Set release thresholds before examining results. Apply gates to each approved task category.

1. Build the application and policy artefact once.
2. Promote changes through dev → test → acc → main.
3. Deploy main to prd.
4. Require a pull request and automated validation for each promotion.
5. Require deployment approval in acc and prd.

In shadow evaluation, the candidate runs without applying proposed changes to shared systems. Approve its data processing and destinations, and isolate its tools.

Deploy infrastructure through versioned NexOps-managed Terraform modules. Deploy Databricks jobs and serving resources through Databricks Asset Bundles from CI/CD pipelines.

Use separate identities, state, workspaces, catalogs and Key Vaults for each environment. Authenticate pipelines through workload identity federation. Use managed identities or federated workload identities for service access.

Require private paths for every service connection. Disable public access where supported. Verify authentication and connectivity before admitting a destination to the experiment.

Define rollback triggers and test the baseline route. If smoke tests, quality checks or telemetry reconciliation fail, stop promotion.

Why It Matters

These release gates address the real effect of routing: it changes the application’s failure patterns, not only its invoice. A candidate can pass output-format checks while producing incomplete analysis or incorrect changes.

The cheapest successful route can differ from the cheapest call. Longer conversations and repeated tool use can outweigh a lower token price.

Our inference cost and latency guide explains why token counts and capacity need joint measurement. Routing adds outcome quality to that decision.

We recommend a small, auditable policy before a learned router. A classifier introduces another source of errors, another evaluation requirement and additional operating cost.

Use Case: Routing Coding Tasks on a Shared Airport Platform

This quality-first approach fits a hypothetical airport operator where many product teams share a Databricks platform.

Starting situation. A central platform team maintains reusable deployment templates and Terraform modules. Product teams submit coding tasks against those repositories, with mandatory review before deployment.

Why the usual approach breaks. One performance-tier model provides a baseline but ignores differences between tasks. A broad switch to a cheaper candidate provides no evidence about which categories remain acceptable.

Applying the change. The team first evaluates documentation-only changes that do not alter deployment behaviour. An approved task enters an isolated sandbox, and the policy selects an authorised destination.

Validation produces a pull request rather than an automatic production change. Infrastructure and access-control changes retain the baseline route and their required reviews.

The harness records task outcomes and correlation identifiers. Governed evaluation tables combine those outcomes with verified gateway and trace records.

What to measure. Compare accepted changes, reviewer corrections, validation failures and later regressions by category and experiment arm. Measure end-to-end latency and cost per accepted outcome, including rejected and abandoned threads.

These are proposed measurements, not reported client results. Incomplete telemetry prevents a release decision rather than counting as evidence of savings.

Decision Guide

The airport scenario suggests a practical rule: choose routing complexity according to the evidence available.

Pitfalls and Misconceptions

Even a small policy can produce misleading results if evaluation, accounting or security boundaries are weak.

A fallback is not a quality guarantee. Semantically wrong answers often return successfully. Escalation needs validated signals, bounded retries and clear termination rules.

An overall average can hide a category regression. Report uncertainty and category-level failures. If evaluation cannot detect the agreed regression margin, collect more evidence before release.

Prompt length is not a reliable risk classification. A short access-control change can require more scrutiny than a long documentation update. Keep risk decisions in trusted policy.

A performance tier is not a security tier. Open SWE warns that coding sandboxes normally receive installation-wide GitHub App access. Repository bindings control routing and preloaded checkouts, not a separate credential boundary.

Open SWE also warns about breaking changes and does not guarantee correctness, stability or compatibility. Its default LangSmith tracing does not establish automatic MLflow coverage.

Tracing creates a sensitive dataset. Inputs, outputs and tool arguments can contain confidential code or personal data. Our governed gateway design covers the broader identity and trace-access boundary.

Usage records are not a complete invoice. Missing costs must remain visible, not become zero. Separate measured spend from estimates and reconcile overlapping records before calculating savings.

Lower median cost does not establish lower total cost. Expensive failures can dominate spend even when the typical thread becomes cheaper. Report total spend alongside cost per successful outcome.

Routing is unsuitable when evaluation is weak. Sparse labels, unstable tasks or expensive correction can favour a fixed baseline. Improve instrumentation and validation before adding a router.

Key Takeaways

1. Establish acceptance criteria and regression limits before comparing model prices.
2. Build representative task categories from traces, including unsuccessful work.
3. Test complete execution paths against held-out tasks and independent checks.
4. Separate application routing from verified platform capabilities and security controls.
5. Reconcile outcomes, latency and spend before promoting a policy through DTAP.
6. Change production traffic only when task-level evidence passes the agreed release gates.