AI engineering / Cost and token optimisation
Cost per task, measured and reduced without losing quality.
The harness around your agent is where cost is controlled and proven. We measure what each task costs, pull the levers that matter and ship a change only when your evaluation set says quality holds.
01 / Where tokens go
Where the tokens in a request go
A single agent request carries more than the question. Each part has its own driver, and each part can be tuned.
| Part of the request | What drives it |
|---|---|
| System prompt and instructions | How long the rules are, and how often they are repeated on every call |
| Retrieved context | How many passages come back and how long each one is |
| Conversation history | How many turns are replayed, and whether they are summarised |
| Tool definitions | How many tools are offered and how much each description says |
| Tool outputs | How much a tool returns beyond what the next step needs |
| Reasoning tokens | The effort setting of the model and the difficulty of the step |
| The answer | The format asked for and how much the answer must contain |
| Retries and loops | Failed calls, repeated tool calls and steps that circle back |
02 / Measure first
Measure before you optimise
You cannot lower what you have not measured. We make cost visible per task, per step and per team before we change anything.
- Cost per task, the number that matters to the business
- Tokens per step, to see which part of the loop is heavy
- Latency per step and per task
- A quality score from the evaluation set, next to every cost figure
- Usage per team and use case, from the Unity AI Gateway system tables
- Inference tables as the record of what was asked and answered
The dashboard is an illustration of the views we build. It shows no real figures.
- Order intake agentWithin budget
Cost per task and quality score shown side by side, per release.
- Reply drafting agentReview
Tokens per step show where the loop is heavy.
- Document extractionWithin budget
Latency and cost per document, per model.
- Team usageTracked
Requests, tokens and latency per team from the system tables.
03 / The levers
The levers that lower cost per task
We pick the levers by what the measurements show. Each one is tested on your evaluation set before it ships.
| Lever | What it does | When it helps | How we prove quality holds |
|---|---|---|---|
| Model choice and swapping | Runs each task on the model that fits it, from a provider API, a licensed model, an open-weight model or a fine-tuned model of your own | Always: it is the first lever, and the harness makes a swap a configuration change | Each candidate model runs the same evaluation set, and cost per task is compared per model |
| Model routing | Sends easy steps to a small model and hard steps to a larger one | Tasks with a mix of simple and difficult steps | The routing rule is decided by evaluation, never by guess |
| Reasoning effort control | Sets how much the model thinks before it answers | Steps that do not need long reasoning | Quality is compared per effort setting on the evaluation set |
| Prompt caching | Puts the stable prefix first: instructions, tool definitions, context | Many calls that share a long, unchanging start | Answers stay identical in the evaluation run, only cost changes |
| Context trimming and history summaries | Keeps only what the next step needs | Long conversations and long tool results | Quality is measured before and after trimming |
| Retrieval tuning | Fewer, better passages per question | Answers padded with loosely related context | Answer quality and source accuracy on the evaluation set |
| Tool output compaction | Tools return only the fields the agent uses | Tools that return whole records | Task success rate stays the same with smaller outputs |
| Structured outputs and early exits | Short, fixed formats, and stopping when the task is done | Tasks with a predictable answer shape | Format validity and task success on the evaluation set |
| Caching repeated answers and tool results | Reuses a result that is still valid | Repeated questions and slow-changing data | Freshness rules are tested with cases that must not be served from cache |
| Batching offline work | Runs work that nobody waits for in batches | Back-office jobs, classification and extraction in bulk | The batch run is scored on the same set as the live run |
| Limits on loops and retries | Caps steps, retries and repeated tool calls per task | Agents that circle or retry without progress | Tracing shows the cap is hit rarely, and the evaluation set still passes |
| Provider choice by latency and throughput | Selects the endpoint that serves the task best | Latency-sensitive tasks and high throughput | Latency and quality are recorded per provider in the same test |
| Self-hosting open-weight models | Serves a model on infrastructure you control | Steady, high volume, or data that must stay where you decide | The self-hosted model must match the evaluation set before it takes traffic |
| Fine-tuning a smaller model | Adapts a small model to one narrow task | A task with stable rules and plenty of examples | The tuned model is compared with the larger model on held-out cases |
Bring your own model, swap it any time Self-hosted and sovereign AI Fine-tuning and other AI solutions
04 / Guardrails on spend
Guardrails on spend
Optimisation lowers the average. Guardrails protect you from the exception.
For coding agents, Omnigent adds policies on cost budgets that track what a session has spent. Open source in alpha; managed version in Beta. Last verified: 11 November 2026. Omnigent: a governed layer above your agents
05 / Proof
How the harness proves each change
Every optimisation runs against the evaluation set. We ship it only when quality holds and cost per task drops, and we log the result per release.
- ChangeA model swap, a cache or a trimmed prompt
- Evaluation setThe same real cases as always
- CompareQuality and cost per task, per model
- GateQuality holds and cost drops
- Release logThe result, per release
Quality and cost are two gates on the same release.
- Model choice and swapping is tested as an A/B run on the evaluation set, with cost per task compared per model
- A shadow run or a traffic split with failover through the gateway shows the effect on live traffic before the switch is final
- The result of every change is logged per release, so the history of cost per task is on record
# Schematic configuration, not a real API
gate:
evaluation_set: real-cases
compare: current-model, candidate-model
quality:
must_not_drop_below: current
cost_per_task:
must_not_exceed: threshold
on_fail: block-release
on_pass: log-result-per-release
MLflow
OpenTelemetry
Databricks
Unity Catalog
06 / Approach
How we engineer it
Five steps, in this order, because each one needs the one before it.
Baseline
We measure cost per task, tokens per step, latency and quality as they are today.
Quick wins
Prompt caching, context trimming and model routing, tested on the evaluation set one at a time.
Structural changes
Smaller or fine-tuned models, or self-hosting, where the measurements show steady volume or a narrow task.
Guardrails
Budgets, rate limits, alerts and cost tests in CI, so the gains stay.
Monthly review
A short review of cost per task, quality and new models, ending in the next change to test.
Governance for AI
One gateway for every model, agent and tool
Every call from teams, agents and coding tools passes through Unity AI Gateway, so access, spend and logs follow the rules of the rest of your data.
- Teams, agents, coding toolsWhoever calls a model
- Unity AI GatewayPermissions, budgets, limits, policies, logging
- Models, MCP servers, toolsDatabricks-hosted and external
How we set it up
- The gateway and its policies are defined as code and deployed by pipeline, never clicked together.
- One service per use case, each with its own budget and an owner who answers for it.
- Inference tables with a retention period and access groups, so logs are useful and not a liability.
- System-table dashboards for tokens, cost and latency per team.
- Coding agents such as Claude Code are routed through the gateway, with repository rules and review gates on top.
- Fallbacks are tested before go-live, by switching a destination off on purpose.
- Smart Routing stays out of production while it is in Beta.
Generally available. Smart Routing is in Beta. Last verified: 10 November 2026. Unity AI Gateway documentation
Further reading
Articles on cost and routing
Written by the engineers who build these systems.
AI Engineering 13 min
Inference on Azure Databricks: What Drives Cost and Latency
Understand inference cost and latency, measure tokens and capacity, and choose batch, pay-per-token, provisioned or self-hosted serving on Azure Databricks.
Read articleAI Engineering 13 min
Model Routing Is a Quality-Control Problem, Not a Cost Switch
Build trace-driven model routing with task-level benchmarks, quality gates and cost-per-success measures, without assuming Azure or Databricks integration.
Read articleAI Engineering 11 min
Unity Gateway: Govern Coding Agents as a Platform Service
Govern coding agents with Unity Gateway: approved models and tools, clear authentication boundaries, trace access, spend ownership and EU processing checks.
Read article
Free check
Take the check
A score from 0 to 100 and a next step, in about three minutes.
07 / FAQ
Questions about cost and tokens
Will a cheaper model hurt quality?
It can, and that is why we test it. A cheaper model runs on your evaluation set next to the current one. If quality holds on your tasks, it takes the task. If it does not, it stays out, or it takes only the easy steps through routing.
Does prompt caching work with our provider?
Most major providers support caching of a stable prefix. We verify the details per provider and model, because the rules differ, and we structure prompts with the stable parts first so caching can apply.
When is self-hosting cheaper?
When volume is steady and high, or when data must stay where you decide. Self-hosting trades usage-based pricing for infrastructure you run, so we compare both on your measured volume before recommending it.
How fast do we see results?
The baseline gives you visibility right away. Quick wins such as caching, trimming and routing are tested one at a time, so each result is on record as soon as its evaluation run finishes.
Next step
What does one task of yours cost today?
Tell us which agent or LLM application you run. We will say what to measure first and which levers are likely to matter.
Currently accepting projects for Q4 2026