Databricks Agent Memory: Starting with a Governed Beta Pilot

An agent that forgets yesterday’s correction creates work for users and platform teams. It repeats discovery calls and requires users to explain the same preferences.

Databricks’ announcement introduces Agent Memory in Beta. It gives agents persistent context across conversations, with managed storage and background consolidation.

We treat that context as application data, with an owner, retention rules and release tests. When agents repeat discovery work, we recommend treating memory as governed application state and proving its value with a synthetic-data pilot before promotion.

What Changed

The Agent Memory announcement introduces persistent storage for facts, preferences and workflows within Agent Bricks, Databricks’ agent developer platform. Developers can read, export and delete entries about each user.

Agent Memory runs on Lakebase, Databricks’ serverless Postgres, provisioned in the workspace and operated by Databricks. Sessions and memories live in the customer’s Lakebase project.

The implementation follows MemCurrent, an architecture that separates conversation records from distilled knowledge. It combines automatic context handling, agent-invoked tools and background consolidation.

Memory retrieval combines keyword matching with semantic search, which finds content through similarity of meaning. Developers can instruct the memory pipeline about what to retain. This changes stored context, not model weights.

A memory pipeline connects one session store to one memory store. It is separate from Lakeflow pipelines. Runs use serverless compute and call the distillation model through Unity Gateway.

Pipeline runs use the identity that created the pipeline. That identity needs access to both stores, table-creation permission in the memory database and permission to query the model. Runs incur serverless general compute and Foundation Model Serving charges.

Databricks reports a threefold task pass rate and 31% fewer tokens in its internal enterprise-task benchmark. These are vendor-reported findings, not NexOps measurements.

Why It Matters for Platform Teams

Memory moves part of agent behaviour from application code into mutable data. The same code and model can produce different answers after a conversation changes that data. Release evidence must therefore identify the memory state used during testing.

Background consolidation also introduces another operational component. Platform teams must assign responsibility for its execution, failures and recovery. We treat it as a data-processing workload.

The opportunity extends beyond personalisation. Repeated investigations can produce reusable context and reduce unnecessary discovery work. Our preferred architecture keeps authoritative business facts in governed data products and uses memory for relevant working context.

Use Case: A Logistics Back-Office Assistant

Consider a hypothetical logistics organisation with operational systems on premises and document processing in Azure. Its automation team supports recurring shipment enquiries under peak-season pressure. Staff repeatedly explain their preferred output format and the steps needed to investigate incomplete paperwork.

The usual approach adds previous conversations to each new prompt. This mixes durable preferences with shipment-specific details and increases prompt size. A separate custom memory store adds extraction, deduplication and retrieval work.

For the pilot, use synthetic shipment cases and a read-only assistant. Approved private tools retrieve shipment metadata and document checklists from governed sources. The application passes only necessary results into managed sessions.

Instruct managed memory to retain each user’s formatting preferences and non-sensitive investigation context. Instruct the pipeline to exclude shipment-specific facts. Current shipment status still comes from the operational source.

Measure the pilot against a no-memory baseline using the same acceptance criteria. Include the complete task, not only the model response.

Implementation

1. Define the memory contract

1. Record the processing purpose and accountable owner group.
2. Define permitted and prohibited content.
3. Define required retention periods for managed sessions and distilled memory.
4. Document deletion and recovery procedures.

For this pilot, permit output preferences and non-sensitive investigation context. Exclude shipment documents, credentials, personal identifiers and unverified business rules.

Derive each user’s actor_id from verified caller identity in trusted application code. Do not trust a request-body user identifier as proof of identity. Manage access through Entra ID groups and privileged administration through Privileged Identity Management (PIM).

2. Generate and review the scaffold

Use an approved build image with a pinned databricks-agentbricks package and a committed dependency lock. For memory pipeline CLI support, use version 0.6.0 or later. Generate the scaffold on a feature branch created from dev.

Review the generated configuration and application code against the current Agent Bricks CLI documentation. Verify supported session and memory declarations, framework adapters and resource bindings before committing the scaffold.

The supplied documentation does not establish configurable retention periods for both stores. Before provisioning, verify that available controls meet the memory contract’s retention and deletion requirements. Do not add undocumented configuration properties.

Keep business logic in src/ and tests in tests/. Run linting, unit tests, dependency checks, licence checks, static analysis and secret scanning on every pull request.

3. Define consolidation as a managed workload

Store reviewed consolidation instructions with the application release. Give them a narrow purpose: retain durable output preferences and non-sensitive investigation context; exclude shipment-specific facts and debugging notes.

Do not run this resource-creation command outside an approved CI/CD deployment path. The memory pipeline documentation shows this syntax for connecting existing stores:

# The stores must exist before pipeline creation.
agentbricks memory pipeline create \
  --session-store support-agent-sessions \
  --memory-store support-agent-memory

The command connects the named session and memory stores. It omits the optional model selection, so the pipeline uses a default model. Review the model choice before release.

1. Start a run with agentbricks memory pipeline run and the returned pipeline ID through the approved automation.
2. Record the returned run resource name.
3. Check the run through the documented REST API.
4. Require SUCCEEDED before evaluating consolidated memory.

The CLI does not report the final run result. A run can fail for one actor while processing others, so check the reported error.

Assign execution, monitoring and recovery to an owner group. Route logs and alerts to central monitoring without recording sensitive conversation content.

4. Test the memory lifecycle

Create a fixed evaluation set with synthetic conversations. Keep the model, tool data and application version constant during comparisons.

1. Run each task without prior memory.
2. Record the answer, tool calls, latency and cost.
3. Add a valid preference or correction in a separate session.
4. Execute consolidation through the verified mechanism.
5. Confirm consolidation completion.
6. Start a fresh session and repeat the relevant task.
7. Evaluate task success and preference retention.
8. Repeat with a different authenticated user.
9. Delete the retained entry through the verified mechanism.
10. Test subsequent retrieval and reconsolidation.

Include conflicting updates, expired information and malicious instructions in the evaluation set. Check that authorised business rules still control the answer. Evaluate the result, not only whether the agent retrieved an entry.

Track the evaluation with MLflow. Record the application commit, model configuration, test-data version, memory-store identity and consolidation execution in the test evidence.

5. Promote through isolated environments

Before provisioning, resolve the availability and deployment questions listed below. Keep infrastructure in pinned NexOps-managed Terraform modules and deploy Databricks application resources through Databricks Asset Bundles in CI/CD.

Use separate identities, stores and state for dev, test, acc and prd. Build the application artefact once and keep environment bindings separate.

Promote through dev, test, acc and main using a pull request for each transition. Require approvals before deployment to acceptance and production.

Use private network paths and disable public access. Authenticate CI/CD through workload identity federation. Keep any necessary secrets in an environment-specific Key Vault.

Maintain a runbook for consolidation failures, incorrect memory and application rollback. Include the affected stores and source sessions in recovery decisions.

Pitfalls and When Not to Use It

Beta access does not establish deployment suitability

The supplied sources do not establish Azure EU availability, supported private-network combinations, authentication compatibility, quotas or configurable retention guarantees. Pipeline documentation identifies compute and model billing categories, but not a complete pilot price.

Before provisioning, confirm Beta access and support for the target Azure EU region. Verify private connectivity, workload identity federation, quotas and retention controls against current Azure documentation. Record the evidence and cost estimate in the design decision.

The announcement says the Agent Bricks CLI can deploy agents with sessions and memory. It does not show deployment commands or establish store-provisioning behaviour. It also does not establish how resource ownership maps to bundles or Terraform.

Verify supported bundle and Terraform integration before approving the deployment design. Document resource ownership, repeat-deployment behaviour and teardown behaviour. We do not use manual CLI deployment as a substitute for the reviewed delivery path.

If no compliant deployment path exists, stop at source review and design. Do not provision a public or manually managed pilot as a workaround.

Consolidation does not establish truth

A newer statement is not necessarily more accurate. Contradiction replacement needs particular care around business definitions and operational procedures. A user correction must not silently override an approved source.

Treat retrieved memories as untrusted context, especially when they contain instructions. Enforce tool permissions outside the model. A remembered workflow must not grant access or authorise a consequential action.

Deleting memory does not delete its source

Managed sessions contain messages, tool calls and results. Excluding content from distilled memory does not remove it from the session record. Apply retention and deletion procedures to both layers.

Incorrect memory can also survive an application rollback because it exists independently of code. Identify affected entries and source sessions before restarting consolidation. Otherwise, the pipeline could recreate deleted content.

Benefits depend on recurring work

The vendor benchmark does not establish fitness for another workload. Token reduction alone cannot demonstrate lower total cost. Use the complete-task measurements defined for the pilot.

We would not add memory to one-off transformations or tasks that already receive complete, authoritative context. We would also avoid persistence without an approved processing purpose.

Key Takeaways

1. Define ownership and permitted content before storing conversations.
2. Separate reusable working context from authoritative business facts.
3. Validate cross-session behaviour with controlled synthetic evaluations.
4. Resolve Beta access and deployment constraints before provisioning.
5. Test deletion and recovery across sessions and distilled memory.
6. Promote only when task results justify the operational cost.