Aimling Lab
Tokenization Lab
Tokenization Lab
See how text becomes tokens in LLMs.
Try an example
Output
Character, word, and token counts update after each run.
Enter text and click Convert to Tokens to inspect the token pieces.
Why This Matters
- LLMs process tokens, not words.
- Different encoders may split the same text differently.
- Hindi, emojis, rare words, and code may produce different token counts.
- Token count affects cost, speed, and context limits.