How Qwen3-0.6B generates a response token by token

From your question to the model's answer

Ask a chat model the question Is OpenAI open?

The answer appears piece by piece: Partly. Some weights are open.

This answer is only five English words, yet the model has to run the same process repeatedly for it. Taking the small open-source model Qwen3-0.6B as an example, this answer consists of 9 pieces in total: the first 8 are text, and the last 1 indicates the end of the answer.

For every piece it writes, the model must run completely once, passing through all 596 million weights. On each run, it assigns a score to each of the 151,936 entries in the vocabulary, and in the end keeps only one. To finish writing this answer, it computed 1,367,424 scores in total.

These pieces are called tokens.

Below, follow a real request through once, from the moment you press Enter to the moment the answer appears on the screen. The tokens and numbers in this article are all taken from Qwen3's real vocabulary. Today's chat models all use the same loop to generate text; the differences lie in model size, vocabulary, and the template that wraps the question.

A language model does not write text directly. Each time it runs, it only scores one round for the next token; a short loop outside turns these scores into text.

Six steps

chat template: The application wraps your question into a fixed format.

tokenize: The formatted text becomes a string of numbers.

prefill: The model reads this string of numbers in one pass and scores the first token of the answer.

selection: Pick one token from the scores.

decode: The selected token is sent back to the model, and the model scores the next token. Steps 4 and 5 repeat until an end token appears.

detokenize: The numbers turn back into text.

Steps 1 to 3 are done only once per request. Steps 4 to 6 are done as many times as there are tokens in the answer.

Step 1: Wrap the question in a format

The model does not see your input question as you typed it. The application first wraps it in the chat template: the fixed text format the model has always used when trained as an assistant. Qwen's format is as follows:

<|im_start|>user Is OpenAI open?<|im_end|> <|im_start|>assistant

<|im_start|> and <|im_end|> are special tokens, that is, single entries in the vocabulary, used to mark the beginning and end of a turn of dialogue.

The last line opens the assistant's turn and then stops here. The model's answer begins on the next line.

What the model reads is the entire formatted text; the question you entered is only one line of it.

Every model family has its own template. With the wrong template, the model sees a format it has never seen in training, and answer quality drops. This is one common reason why a self-deployed model performs worse than the official demo.

Step 2: Text becomes numbers

The model can only compute on numbers, so text must first become numbers. This step is done by the tokenizer.

The tokenizer relies on the vocabulary: a fixed list of text pieces, each with a line number called a token ID. Qwen3's vocabulary has 151,669 entries.

The tokenizer splits the formatted question into pieces that already exist in the vocabulary, then replaces each piece with its number. Our question becomes 13 tokens:

151644 872 198 3872 5264 15469 1787 30 151645 198 151644 77091 198

From this step on, the model sees only numbers, not text.

The splitting result has several characteristics:

A token can be a whole word, part of a word, punctuation, or a special marker. OpenAI is two tokens: ␣Open and AI.

The space before a word belongs to that word's token. ␣open (1787) and open are two different entries.

English text is being prepared.

A newline is also a token (198), used three times in the template.

The vocabulary is fixed before training begins. It does not change while the model learns, and it does not change when the model answers questions.

Step 3: The model reads the entire prompt in one pass

The weights are the model itself: 596 million numbers determined during training, unchanged when answering questions. The 13 numbers pass through three stages.

embedding. Each number retrieves the corresponding row from a table. This table has 151,936 rows, each with 1,024 numbers. The tokenizer uses 151,669 of these rows; the rest are unused padding rows. This one table alone has 155.6 million weights, accounting for 26% of the entire model.

28 layers. The 13 vectors then pass through 28 layers. In each layer, every position looks at the positions before it (this is called attention), then updates its own vector. The 13 positions pass through together, requiring only a single run. This stage is called prefill.

LM head. Only the vector at the last position continues onward. The LM head compares it with every entry in the vocabulary, giving each entry a score—151,936 in total—called logits. In Qwen3-0.6B, the LM head directly reuses the embedding table.

Along the way, each layer stores two vectors for each position: key and value, together called the KV cache. Later tokens will use them again. Once stored, the prompt does not need to be computed a second time.

A single run gives each token in the vocabulary a score. The highest score is only a candidate; it has not yet been selected.

Step 4: 151,936 scores become one token

Logits are raw scores. Getting a token from them takes only a few arithmetic steps and involves no weights:

softmax: turn the logits into probabilities, all positive, summing to 1.

temperature: before softmax, divide the logits by it. Less than 1 concentrates the probability on the highest token; greater than 1 flattens the distribution.

top-p: keep the smallest set of tokens whose probabilities sum to p (here 0.9), and discard the rest.

greedy decoding or sampling: the former takes the token with the highest probability, the latter randomly draws one according to the probabilities.

In our illustration, after top-p three tokens remain: Part at 57%, ␣Open at 29%, and No at 14%.

greedy decoding writes Part every time. sampling writes Part most of the time, sometimes ␣Open, and occasionally No.

These settings are determined by the application or the API call, and have nothing to do with the model's weights.

The same set of weights can give the same answer every time, or a different answer every time; the only difference is how the final choice is made.

Step 5: The loop

The chosen token is appended to the end of the sequence, and the model runs again. This stage is called decode.

During decode, the model does not reread the entire prompt; only the newest token passes through 28 layers. In each layer, it reads the existing keys and values in the KV cache—one set for each previous position—and stores its own set as well.

The LM head scores the next token, selects one, and sends it back to the model. This repeats.

The loop stops when the model selects the end token <|im_end|>, or reaches the length limit set by the application. In our example, prefill gives Part, and the next 8 decodes give, in order, ly, ., ␣Some, ␣weights, ␣are, ␣open, ., and finally <|im_end|>. 9 tokens, 9 runs.

prefill processes the entire prompt at once; decode processes only one token at a time, and each time must reread all of the model's weights.

Step 6: Numbers back to text

The output IDs are looked up in reverse along the same vocabulary:

each ID is converted back to its text fragment;

the end token is removed;

leading spaces stay in front of their respective words;

the fragments are joined into the complete answer.

The application does this step as each token arrives, so the answer appears piece by piece.

Some characters, including many Chinese characters, span more than one token. The application temporarily holds incomplete bytes and displays the character only once it is complete.

The same vocabulary is used twice: on the way in, text is turned into IDs; on the way out, IDs are turned back into text.

What this loop explains

Many phenomena noticed when using chat models can be explained by this loop.

The answer appears piece by piece. The model produces one token per run, and the application displays each one as it arrives.

The same question gets a different answer each time. Chat applications usually use sampling. Lowering the temperature, or switching to greedy decoding, makes answers more stable.

There are two kinds of waiting. When the prompt is long, the first character comes out slowly, because prefill has to process all of it. When the answer is long, overall completion is slow, because decode must run separately for each token. Speed metrics are also split into these two: time to first token, and tokens generated per second.

Counting letters is hard. When asked How many r are in strawberry?, Qwen3 treats ␣strawberry as a single token, ID 72600. There is not a single letter in the input, so the model can only answer based on spelling knowledge about token 72600 learned during training. If there is no space in front, strawberry is split into three tokens: str, aw, berry.

Quotas and prices are calculated by token. Context length, API pricing, and rate limits are all measured in tokens; character or word counts can only be roughly converted. How many tokens a sentence takes depends on the language and the vocabulary.

The model does not learn during a chat. The weights never change. Your conversation exists in only two places: the token sequence the application sends each time, and the KV cache computed from it.

Every token in the answer comes from the same loop: score the entire vocabulary, pick one, append it, and go again.

Summary

The application wraps the question in the chat template, and the tokenizer then turns it into token IDs using a fixed vocabulary.

prefill sends the entire prompt into the model at once, fills the KV cache, and produces logits: one score for each entry in the vocabulary.

softmax, temperature, top-p, plus greedy decoding or sampling, turn the logits into one token. This step involves no weights.

decode sends each new token back into the model, running once per token, until the end token appears.

detokenize turns the IDs back into text, displaying one token at a time as it arrives.