Before a large language model can do anything useful with your prompt, it has to convert raw text into numbers. Understanding this conversion explains many of the quirks you will meet later, from strange spelling mistakes to surprising costs on long documents.
What a token actually is
A tokenizer splits text into small chunks called tokens. A token might be a whole word, part of a word, a punctuation mark, or a space. Common English words are often one token, while rare words, code, and other languages get split into several pieces, which is why identical-length texts can have very different token counts.
From tokens to embeddings
Each token is mapped to an embedding, a long list of numbers that captures meaning learned during training. Words used in similar contexts end up with similar embeddings, so the model can treat synonyms and related ideas as neighbors in this numeric space rather than as unrelated strings of letters.
Why this matters for prompting
Because models see tokens rather than characters, tasks like counting letters or reversing words are harder than they look. Token counts also drive pricing and context limits, so verbose prompts cost more and leave less room for your documents and the model’s answer.
Action Step
Paste three different passages into a free online tokenizer tool and compare the token counts to the word counts. Note how code, numbers, and unusual names split into multiple tokens, and estimate how many tokens your typical working prompt uses.
This lesson is educational content, not professional advice. Results depend on your own situation and effort; no outcome is guaranteed.