A chatbot can't read letters like you do. First it chops your text into little pieces called tokens — and here's the wild part: tokens are what you pay for. Let's watch the chopper work on anything you type.
Every colored chunk below is one token. Notice: common words are usually 1 token, but long or rare words break into pieces. Type your name, a big number, or an emoji and see what happens!
🟦 word piece 🟨 digit 🟪 emoji / other language ⬜ symbol. The little number under an emoji is how many tokens it secretly costs!
Tap each button — it drops that text into the chopper above. Guess what'll happen before you tap. Some tiny things cost a shocking number of tokens!
A phrase appears. Guess how many tokens it is with the slider, then hit Reveal. Get within 2 and you score. Nail 3 phrases to win the badge!
your guess (tokens)
Our chopper is a mini version of the real thing. Here's how tokenizing grew up — from dead simple to what powers real AI:
Just cut at every space = "words". ✅ Super simple. ⚠️ Every new or misspelled word is a mystery, and the dictionary gets huge.
Break rare words into common pieces ("un + believ + able"). ✅ Handles any word. ✅ Small vocabulary. ⚠️ Numbers & emojis split oddly.
Learn the best pieces from billions of words of text. ✅ Efficient. ✅ Works in every language. ⚠️ You have to know its quirks to save money.
Everything above was a teaching tokenizer. This is the actual GPT-2 tokenizer — the real BPE that powers real models — running 100% on your own device, no server, and no internet once it loads. Load it once (~2 MB) and compare the counts!
You met the token — the tiny piece of text that AI reads and charges for. You saw that tokens aren't words, and that numbers, emojis, and other languages tokenize in surprising ways. This is the foundation of everything about AI cost.