Memory is the token optimisation nobody is doing
Prompt caching and model routing get the attention. The larger win is usually not sending the context at all.
The short answer: most teams optimising token spend are compressing prompts and routing models while re-sending the same project context at the start of every session. Persistent memory removes that spend entirely rather than making it cheaper, and it is typically a larger saving than caching and routing combined.
Where the tokens actually go
Open any token-optimisation guide and you get the same five levers: model routing, prompt compression, caching, batching, and shorter outputs. All real. All working on the same assumption — that the context needs to be sent, and the job is to send it more cheaply.
Watch what a working session actually looks like. You open a new conversation. You explain the stack. You paste the file. You remind the assistant that you prefer diffs over whole files, that the database is SQLite now and not Postgres, that Ana owns the API. Then you ask your question.
Everything before the question is re-transmission. It was true last week and it is true now, and you paid for it both times.
Putting a number on it
Take a modest case: 2,000 tokens of standing context, 12 sessions a week. That is 24,000 tokens weekly, about 1.25M a year, purely to restate things that never changed. On a mid-tier model that is real money, and on an agentic workload making dozens of calls per task it multiplies.
The comparison that matters is not against zero. It is against the cached version of the same waste. Prompt caching discounts repeated context heavily, which is genuinely useful — but a cached token is still a token you chose to send. Memory changes the question from “how cheaply can I resend this?” to “why am I sending it?”
You can measure your own with the token estimator: paste the preamble you find yourself repeating and multiply by your session count.
Why retrieval beats re-injection
The naive fix is to dump everything you know into the prompt. That fails in both directions: it costs more than the preamble it replaced, and long contexts degrade retrieval quality — the relevant fact competes with everything else you loaded.
Retrieval sends what the current question needs. A well-built memory answers “what did we decide about the database?” with two or three facts and their provenance, not with your entire project history.
The pattern that saves the most: find, then fetch
The single largest waste inside a memory system is loading twelve full memories in order to read one. A brief search that returns subjects and identifiers costs roughly a fifth of the tokens of one returning bodies. You scan the list, then fetch the one body you need.
This is invisible in your bill because nothing errors. The answers are fine. You simply pay several times over for context the model glanced at and discarded.
What this costs you when it is missing
- Re-transmission — the preamble, every session.
- Re-derivation — the assistant re-reads and re-summarises the same files because last week's conclusion is gone.
- Correction loops — it repeats a mistake you already corrected, and you spend tokens correcting it again.
- Stale context — it confidently uses a fact that changed, and the whole exchange is wasted.
The last two are the expensive ones, and neither shows up as a line item.
Doing it without a vendor
Nothing here requires a product. A folder of Markdown notes, an embedding model, and a retrieval step gets most of the benefit. What you need is a discipline about what goes in, which is the harder half — covered in making AI memory actually work.
Varven is one implementation: plain Markdown on your own disk, replicated to a cloud your assistant can read when your machine is off, and readable by any client that speaks the Model Context Protocol.
Common questions
Is memory cheaper than prompt caching?
They solve different problems and compose well. Caching discounts context you send repeatedly; memory removes the decision to send it. Use caching for the system prompt that genuinely must be present every call, and memory for facts that only some questions need.
Does memory increase token usage?
It can, if it retrieves badly. A system that returns ten memories per query costs more than the preamble it replaced. Retrieval quality is the whole game — measure tokens retrieved per useful answer, not memories stored.
How much can I expect to save?
Highly dependent on how much standing context you restate. Teams with heavy project preambles see the largest wins; one-off stateless queries see none. Measure your own preamble before believing anyone's percentage, including this page's.