What Are Tokens in AI? Explained Simply, with Examples

How text turns into tokens

A neural network doesn't see letters. Before it reads your message, a special program called a tokenizer chops the text into pieces from its vocabulary and replaces each one with a number. The model then works with a string of numbers, and builds its answer the same way, piece by piece.

Common words make it into the vocabulary whole and become a single token. Rare ones get split into parts: "tokenization" might turn into "token" and "ization." Punctuation and spaces count too. So a short "Hey, how's it going?" isn't four words, it's six or seven tokens.

Every model has its own vocabulary. The same phrase might take five tokens in one model and eight in another. "How many tokens per word" only makes sense within a single model.

Why Russian text is "heavier" than English

In English, a word takes a little over one token on average. In Russian, modern models use about two tokens per word, and older models up to three. There are two reasons:

• Vocabularies are built mostly from English text. English words get more room, Russian words get less, so they're split into pieces more often.
• A Russian word has lots of forms: dom, doma, domu, domami (all variations of "house"). Each form is a separate entry, and not all of them make it into the vocabulary whole.

What that means in practice: the same conversation in Russian takes up more of the model's memory and, if you pay per token, costs more.

Prompt, context, and window

When you send a message to a character, the model doesn't get just your one line. The service assembles a whole package:

1. The service's instructions: how to reply, what not to do.
2. The character description: personality, way of speaking, backstory.
3. Memory about you: your name, important facts.
4. The chat history.
5. Your new message.

All of that together is called the context, and the limit on how much context the model can read at once is the context window. The window is measured in tokens. In modern models it's huge, hundreds of thousands or even a million tokens, but services rarely use all of it: every token read costs money and time.

Why a character forgets a long conversation

Here's a key detail: the model rereads the entire context at every step. At message one hundred, it reads the first ninety-nine all over again. So a long conversation gets heavy, and services do something about it. That leads to three reasons for forgetfulness:

• History gets trimmedWhen the chat no longer fits, the oldest messages fall out of the window. The model simply can't see them anymore.
• The middle is harder to seeResearchers noticed that models remember the beginning and end of a long text better and miss the middle. The effect even has a name: "lost in the middle."
• Summaries lose detailsMany services compress the older part of a conversation into a short summary. The color of a scarf or the dog's name might not make it in.

How to help the model remember

• Repeat important things closer to the end. If you agreed early on that the character calls you "captain," remind it once the conversation has grown.
• Every so often, ask for a recap: "list what we've agreed on." You'll also see what the model has already lost.
• Start a new topic in a new chat. A short recap as your first message works better than an endless feed.
• Keep it focused. The less filler, the more of what matters fits in the window.

How to write a request so the model gets what you want on the first try is covered in how to write a prompt. And what to do if answers get muddled or stop coming altogether is in when the AI won't respond.

How it works on my-idol.ru

Characters reply in English or Russian, whichever you choose, and remember the conversation, and the tips above work here too: remind them of what matters and start a new story in a new chat. More on what a character remembers and how in AI character memory.