The first time Claude Code told me how many "tokens" I'd used, I had no idea what I was counting.
A token is a small chunk of text. AI doesn't read word by word. Before the AI sees your message, it's chopped into tokens, and the AI works with those tokens, not whole words. A short, common word like "the" is usually one token. A longer or less common word gets split into pieces.
Medical words, counted
I ran some medical terms through OpenAI's free tokenizer, set to GPT-4o. You can paste in your own words and try it.
| Word | Tokens | How it splits |
|---|---|---|
| fever | 1 | fever |
| pneumonia | 1 | pneumonia |
| troponin | 3 | tro · pon · in |
| acetaminophen | 3 | acet · amin · ophen |
| bronchiolitis | 4 | bron · chi · ol · itis |
| intussusception | 4 | int · uss · us · ception |
| pheochromocytoma | 5 | phe · och · rom · ocyt · oma |
Common words get one token. Rarer words get chopped up, and the pieces don't follow the Latin and Greek roots we learned in medical school. These are OpenAI's countsEach AI company splits text its own way. Claude's tokenizer isn't public, so its counts will differ a little..
The rough math
Easy way to remember it: a token is about three-quarters of a word, or 4 tokens for every 3 words. So 1,000 tokens is about 750 English words. The exact count varies by model, and newer Claude models count more tokens for the same text. Even a short blog post can use more than a thousand tokens. Uploading a 30-page PDF can use tens of thousands.
Spaces, punctuation, and code count too. So does everything the AI writes back to you.
Why a beginner should care
1. It's how the AI measures its memory. An AI chatbot can only work with so many tokens at once. That limit is called the context window. More on it in an upcoming AI 101 post.
2. It's how API use is billed. When a program uses an AI through its API, the bill is usually counted in tokens: the ones you send in, plus the ones it sends back.
If you're on a monthly plan like Claude Pro, you pay a subscription, not a charge per token. Your usage limitThat's the message telling you you've hit your limit and when it resets. still runs out faster with long messages, big attachments, and long conversations.
Tokens measure the text an AI reads and writes. They set how much it can hold at once and what API use costs.