Token estimator
Paste text. Get an estimated token count, the share of a context window it fills, and what embedding it would cost.
How the estimate works
Modern models use byte-pair encoding, which splits text into sub-word pieces. Common English words are one token; rare words, punctuation and whitespace-heavy code cost more. Without shipping a megabyte of vocabulary to your browser, the practical approach is a calibrated heuristic: count words, punctuation and whitespace runs, then adjust for content type.
Expect this to land within about 10% for English prose, and
wider for code and other languages. If you need exactness, run the model's own
tokenizer — tiktoken for OpenAI, or the tokenizer shipped with
the model on Hugging Face.
Nothing you paste leaves your browser. There is no request, no logging, and no server involved — the arithmetic happens on this page.
Why token counts matter for memory
Every fact your assistant retrieves costs context, and context is finite. A memory system that returns twelve full notes when one would do has not helped you — it has spent your window and pushed the actual conversation out of it.
This is why Varven separates finding which memory from reading what it says: a brief search returns subjects and ids for roughly a fifth of the tokens, and you pull the one body you need. See how Varven works.
Common questions
How many tokens is a word?
English prose averages about 1.3 tokens per word, or roughly four characters per token. Code runs higher because identifiers, brackets and indentation fragment into more pieces.
Why do different models report different counts?
Each model family has its own vocabulary. The same paragraph can differ by 10–20% between GPT, Claude and Llama tokenizers. Estimates here are tuned to that middle ground.
Does whitespace count?
Yes. Indentation and blank lines are real tokens, which is why minified JSON is cheaper than pretty-printed JSON with the same content.