A token is a unit produced by a model’s text encoding. Use the right counting method for the selected model, and distinguish the text fixture you count from the complete request you eventually send.
Definition with a live example
OpenAI’s tiktoken uses byte pair encoding to turn text into a sequence of token identifiers. Official documentation.
Tokens are not a synonym for words. A tokenizer can split a word, combine a common sequence or assign units to punctuation and whitespace. A character count measures the written string; a word count follows a different text rule. Each is useful for a different purpose, and none should be relabelled as another simply because their values often move together.
A safe way to learn is to keep a short fixture unchanged and compare available encodings. Include ordinary prose, a code identifier, punctuation and a repeated string. Observe the count under each method while preserving the actual text. If you edit the fixture at the same time, you can no longer isolate the effect of the tokenizer choice.
Count tokens locally
Your text and estimates stay in this browser. No API requests are sent to model providers.
Loading verified model records…
The live tool distinguishes supported exact encoding from a labelled approximation. An exact count applies to the supplied text and selected encoding. It does not prove that every new model uses that encoding or that unpasted request material is included. The method and scope should accompany any count copied into an estimate.
An approximation can be useful for early sizing. Its precision is limited by the heuristic, even if the output is displayed as a whole number. Keep that uncertainty visible and replace it with a provider-specific count or observed usage when the decision is close to a capacity or budget boundary.
How it affects cost / limits / results
Input tokens describe material supplied to the model operation. Output tokens describe generated work under the provider’s accounting. A full application request may include system or developer instructions, examples, conversation history and tools as well as the current question. Count the assembled shape when forecasting rather than assuming the visible text box is the entire input.
A configured output cap is not observed usage. It bounds generation according to the request contract, while the actual response can consume a different amount. Keep the cap, completion reason and returned usage as separate fields. That distinction explains why multiplying a maximum by every request can differ from measured application spending.
Cached-input accounting is another category. Repeated material may qualify for reuse under a provider’s rules, but ordinary text repetition does not guarantee a cache hit. Preserve the applicable request structure and actual returned category before assigning a cached share to a workload.
Use the cost calculator to combine supported input and output assumptions with model rates. If the application uses media or tools with other units, retain those categories separately instead of converting them into a guessed text-token total.
Token volume also affects capacity planning. A small number of large requests can create a different pressure pattern from many tiny calls. Preserve request rate and token workload separately in your telemetry. When a request fails, inspect the specific limit rather than assuming a high token count explains every rejection.
Provider differences
OpenAI provides tiktoken for supported encodings. Use the model-appropriate method and distinguish plain text counting from any additional request representation.
Anthropic offers a token-counting endpoint with its own request shape and limitations. A generic local text estimate should not be presented as an exact Claude account measurement.
Google documents Gemini token counting and input-type accounting. Use its relevant method when a Gemini-specific measurement is needed.
For cross-provider comparison, begin with the same underlying task, then apply each appropriate counting method. Reusing one count everywhere can support a rough sensitivity exercise, but label that assumption. For a final decision, retain the provider’s returned usage from representative accepted tasks.
Keep the original safe fixture so a prompt revision can be evaluated later. Token savings are useful only when the revised request still contains the evidence and instructions needed for an acceptable result. A shorter prompt that removes a critical rule can lower input while increasing failed or repaired tasks.
Common mistakes
Do not multiply a word count by an assumed universal constant and describe the result as exact. Tokenization differs by method and text structure. Keep the tool’s estimate label when sharing the value.
Do not omit hidden application context from a budget. Inspect what the worker actually sends, including repeated history and tool definitions. A fixture that represents only the last message may materially understate the real request shape.
Do not use a text count to price an image or audio attachment without documented accounting. A path to a media file is not the media itself, and its written filename does not describe the billable input.
Do not optimize solely for fewer tokens. Compare accepted-task outcomes before and after a prompt reduction, and include repair calls. The useful objective is an efficient valid result rather than the smallest possible request.
Last verified · Source ↗
Frequently asked questions
Are tokens words?
Is a character-based estimate exact?
Does an output cap equal usage?
What else can be in input besides the question?
Can one token count be reused across every provider?
How do I judge prompt shortening?
Sources
Last verified · Source ↗