Private AI utility
Token Visualizer
Explore a local approximation of how text becomes model tokens. Everything runs in your browser; your prompt is never uploaded.
Illustrative local token chunks (not a provider tokenizer)
Start typing to visualize text segments.
How it works Token Visualizer
The visualizer splits text into words, whitespace, numbers, and punctuation. It teaches the idea of segmentation without claiming to reproduce a provider’s proprietary tokenizer.
Example
Try ‘ChatGPT’ or a line of code to see the browser’s illustrative text segments.
Frequently asked questions
- Why do the chunks differ from provider tools?
- Every model family can use a different tokenizer and message-formatting rules.
- Does it send text away?
- No. Segmentation happens in the page.