Characters vs Words vs Tokens

Characters, words, and tokens are three different ways of measuring text, and confusing them can make AI cost estimates inaccurate.

What consumes the limit

A character is an individual letter, digit, punctuation mark, symbol, or space. For example, “Hello!” contains six visible characters.

A word is a human-readable linguistic unit. “Hello from the AI” contains four words.

Leave room for a useful response

A token, however, is a unit created by an AI model's tokenizer. Tokens can represent whole words, pieces of words, punctuation, spaces, or other text patterns.

Because of this, there is no universal conversion such as “one word equals one token.”

Check the model-specific constraint

For ordinary English prose, a commonly used approximation is that one token represents roughly three to four characters and that 100 words may require around 130 tokens. Treat those numbers only as estimates. Code, numbers, URLs, unusual vocabulary, other languages, and repetitive structured data can tokenize differently.

For example, source code often includes braces, operators, variable names, indentation, and punctuation. A 1,000-word technical document and a 1,000-word article therefore may not consume exactly the same number of tokens.

Practical checks before you proceed

Why does this matter?

AI APIs normally charge by tokens, not words. Model context windows are also expressed in tokens. If you're estimating how many documents can fit into a prompt, counting words alone can therefore be misleading.

Practical checks before you proceed

For casual estimates, character and word counts are useful. For billing forecasts or strict context-window management, use the tokenizer designed for your selected model whenever possible.