Reinforcement Learning Terminology for LLM Readers: How a Single Score Is Distributed to Each Token

How a score is distributed to each token

In January 2025, DeepSeek released DeepSeek-R1-Zero. It performed reinforcement learning (RL) training directly on the base model, without any supervised fine-tuning in between.

Its reward comes from rules: one rule checks whether the final answer is correct, and another checks whether the reasoning process is written between the think tags.

After several thousand steps of RL training, its pass@1 on the AIME 2024 math competition rose from 15.6% to 71.0%.

During training its answers also grew longer: the model learned to use more tokens to reason.

The rule check gives only one score for the entire answer, while an answer may contain several thousand tokens. What training needs to adjust, however, is the probability of each token. Reinforcement learning for LLMs, in the end, answers one question:

The judge gives only one score for the entire answer. During training, which tokens' probabilities should be increased, and which tokens' probabilities should be decreased?

Reinforcement learning has its own set of terms:

policy, state, action, reward, return, value, advantage, baseline, TD error, GAE, clipping, KL

Each term corresponds to part of the answer to this question. This article uses one example to explain them one by one. The next two articles, on preference alignment and RLVR (reinforcement learning with verifiable rewards), will use these terms directly. This is the fourth article in the post-training series; the previous one introduced LoRA and QLoRA.

The example used throughout

The question is Is OpenAI open? The name says open. The answer may be partially open.

Suppose the model has already written OpenAI is, and now needs to choose the next token. It is most inclined to choose the most obvious one:

open, probability 0.5, followed by It's in the name. The judge gives an average score of 2 (out of 10).

closed, probability 0.3, missing the open-weights models. The judge gives an average score of 5.

partly, probability 0.2, followed by some open weights. The judge gives an average score of 9.

The loop: agent, environment, reward

In reinforcement learning, the agent takes an action, the environment gives feedback, and the reward measures how good the action is. Mapped to an LLM:

The agent is the model. The model's probability distribution over the next token is called the policy, written π_θ, where θ is the model weights. Training changes θ, and thus changes the policy.

The environment is everything outside the model: the prompt the model receives, and the judge that scores the answer.

The judge is the part of the environment that scores the complete answer. It can be a reward model trained on human preference data, a rule checker like the one in R1-Zero, a unit test that runs generated code, or another LLM that scores according to a rubric. This article refers to all of them collectively as the judge, so that the terms in this article apply to all these cases.

The judge counts as part of the environment because the model cannot change it: during RL training the reward model is frozen, and like the prompt set it does not change.

One round of the loop works like this: the environment gives a prompt, the model writes an answer, the judge returns a score, and the score is used to update the policy. The next round continues with the updated model.

state, action, trajectory

Within an answer, reinforcement learning proceeds token by token.

The state is everything the model sees before writing the next token: the prompt plus the tokens already written. In the example, that is Is OpenAI open? plus OpenAI is.

The action is the next token. The policy assigns a probability to each candidate token (open 0.5, closed 0.3, partly 0.2), and then one is selected by sampling according to the probabilities.

The selected token is appended to the state, forming a new state. This repeats until the answer ends. A complete answer is called a trajectory, also called a rollout or an episode.

An LLM is just a policy: the prompt plus the tokens already written is the state, and the next token is the action.

reward and return

Suppose the model samples open, and the completed answer is OpenAI is open. It's in the name. The judge reads the entire answer and gives a score of 2.

The reward is the score obtained at each step. In reinforcement learning for LLMs, every token's reward is 0 except the last token, which receives the judge's score. This kind of reward is called a sparse reward, also called a delayed reward.

A half-written answer has no definite score: what follows OpenAI is could be written well or badly. To score each step, one needs a process reward model trained on step-by-step labeled data.

The return is the sum of all rewards from a given token to the end of the answer. General reinforcement learning multiplies later rewards by a discount factor γ less than 1, discounting rewards further the later they occur, which makes sense for tasks with very long durations.

LLM answers end, and the score belongs to the entire answer, so reinforcement learning for LLMs takes γ = 1, with no discounting. If γ were less than 1, earlier tokens would receive less of the score, even if they determine the entire answer (as open does in the example); moreover, the shorter the answer, the less the discount, so the model would be biased toward writing short answers.

With γ = 1 and only one reward at the end, the return at every position equals the final score: OpenAI is 2, is is 2, open is 2, and name is also 2.

So looking only at the return, one cannot tell which token is good and which token is bad. Every token in this answer receives the same 2. The later terms all exist to distribute this score across different tokens.

Sparse reward has another cost: a judge that looks only at the outcome cannot see how the outcome was reached. OpenAI is partly. It's in the name. has the right conclusion but the wrong reasoning, and a judge that looks only at the conclusion will give it full marks. Guessing correctly by luck, having wrong reasoning but a number that happens to be correct, and writing text that becomes harder and harder to read can all likewise earn the score.

value, Q, and advantage

Return to the state OpenAI is. Let the model continue writing from here many times, and hand each completed answer to the judge for scoring. The figure shows ten of these: according to the policy's probabilities, 5 start with open, 3 start with closed, and 2 start with partly. Even with the same start, the rest is written differently, so the scores vary:

The 5 answers starting with open score 3, 1, 2, 1, and 3 respectively;

The 3 answers starting with closed score 5, 4, and 6 respectively;

The 2 answers starting with partly score 10 and 8 respectively.

From these ten scores, three quantities can be computed. In the notation, s denotes the state, here OpenAI is; a denotes the action, that is, the next token, here one of open, closed, and partly.

action value Q(s, a): the average score obtainable by first writing token a in state s and then continuing as usual. Group the answers that start with the same token and average within each group: Q(open) = (3 + 1 + 2 + 1 + 3) / 5 = 2, Q(closed) = (5 + 4 + 6) / 3 = 5, Q(partly) = (10 + 8) / 2 = 9. The scores of 2, 5, and 9 given to the three kinds of answers in the earlier example are exactly these three averages.

value V(s): the average score obtainable in state s when the next token is not specified and the model continues writing in its usual way. Averaging all ten scores together: (3 + 1 + 2 + 1 + 3 + 5 + 4 + 6 + 10 + 8) / 10 = 4.3. It also equals the policy's probabilities weighting the three Q values: V = 0.5 × 2 + 0.3 × 5 + 0.2 × 9 = 4.3.

advantage A(s, a) = Q(s, a) − V(s): the average score for writing token a in state s, minus the average score when no token is specified—that is, how much higher or lower writing a is compared with the model's usual level. For example, A(open) = Q(open) − V = 2 − 4.3 = −2.3, meaning that an answer that starts with open scores 2.3 points below average; similarly, A(closed) = 5 − 4.3 = +0.7, and A(partly) = 9 − 4.3 = +4.7.

The figure plots all three together. The ten sampled answers fall on their respective scores: the points on the same branch collapse to their average, giving that branch's Q; averaging all ten points together gives V. The state is drawn at the height V = 4.3, and the arrows from the V line to each Q are the advantages.

What actually drives training is the advantage. It has four properties worth remembering.

It answers a question that the raw score cannot. The judge gave open a score of 2. Whether 2 is good or bad depends on how many points the model usually gets at this position. The reward is an absolute score; the advantage compares it with what the model is expected to get in this state.

The sign is the direction, and the magnitude is the size. If A is greater than 0, raise the probability of this token; if A is less than 0, lower it. The larger |A| is, the bigger the adjustment.

It is relative to the current model. The closed answer is worse than partly, yet it gets +0.7, because it is above the model's current average score of 4.3. Once the model improves and its average score exceeds 5, the advantage of the same closed answer will become negative.

Weighted by probability, its average is 0. 0.5 × (−2.3) + 0.3 × 0.7 + 0.2 × 4.7 = −1.15 + 0.21 + 0.94 = 0. A single update cannot raise the probability of every token; it can only shift probability from tokens below average to tokens above average.

The advantage expresses: at this position, how much better writing this token is than the model's usual way of writing.

A major difference between algorithms is where V comes from. PPO trains an additional network (the critic) to predict V. GRPO, which R1-Zero uses, samples a group of answers to the same prompt and uses their average score in place of V. The next post will introduce these two algorithms.

baseline

During training Q is unavailable; we can only see individual sampled answers and their scores R. So in practice we estimate each answer's advantage by subtracting a baseline from R; the baseline is usually taken to be V. In the figure, the five open answers score 3, 1, 2, 1, 3, so R − V is −1.3, −3.3, −2.3, −3.3, −1.3 respectively, whose average is −2.3—exactly equal to Q − V.

In the example the judge's scores are all positive. If we count from 0, every sampled answer gets pushed up, open included. It is merely pushed up less than the others, and the correct direction only emerges after many samples cancel each other out. Counting from V, each sample already points in the correct direction: answers below average are pushed down, and answers above average are pushed up.

As long as the baseline does not depend on which token was sampled, subtracting it does not change the average update—only the noise. In the simulation for this example, using 0 as the baseline and using V as the baseline give exactly the same average update; with V as the baseline, the variance is about one fifth of the former.

The baseline changes the noise of the update; the direction of the update stays the same.

GRPO's within-group average score and PPO's critic are both baselines in this sense.

credit assignment: TD error and GAE

The advantage above concerns only one decision—the token after OpenAI is. A real answer has many tokens, and each token must get its own share. Deciding which tokens the score should be attributed to is called credit assignment.

Follow the sampled answer and watch how V changes. Before writing open, V = 4.3. After writing open, the expected score becomes 2, because answers that write open score about 2. From then until the end, V no longer changes.

TD error (temporal-difference error) measures this change at each token: δ = r + V(next state) − V(current state). It is 0 at every position other than open, and at open it is 0 + 2 − 4.3 = −2.3. Whichever token the expected score drops at, that token gets the blame. It's in the name bears no responsibility, because by the time it is written the outcome is already determined.

Credit goes to the token where the expected score changes.

This requires V to be accurate, whereas in practice V is a learned estimate. GAE (generalized advantage estimation) introduces a parameter λ between 0 and 1. When λ = 0, each token's advantage is its own TD error, and all the blame falls on open. When λ = 1, each token's advantage is the final score minus the V at that token, so OpenAI, is, and open each get −2.3. This assigns blame to tokens that did nothing wrong, since every answer starts with OpenAI is; with enough samples, this noise averages out. λ = 0 relies entirely on V, while λ = 1 relies only on the final score; in PPO, λ is often around 0.95.

GRPO has no per-token V. It makes all tokens of an answer share the same advantage—the result of comparing that answer's score with the other answers in the same group. In our example, all ten tokens get the same negative value, including It's in the name.

policy gradient

Once each token has an advantage, the update rule can be stated in one sentence: for tokens with positive advantage, raise their log probability; for tokens with negative advantage, lower their log probability; the size of the adjustment is proportional to the advantage. This is the policy gradient. In our state, the change in each candidate token's logit is proportional to π × A.

Updating exactly two steps according to the expected gradient (step sizes are illustrative), the probabilities go from 0.50 / 0.30 / 0.20 to 0.15 / 0.36 / 0.49, then to 0.04 / 0.11 / 0.86; V rises from 4.3 to 6.5, then to 8.3. In the first step the probability of closed rises, because 5 is above 4.3. Once V exceeds 5, closed is below average and its probability starts to fall. The advantage is always relative to the current model; as the model improves, the standard rises with it.

exploration

An update can only use answers the model actually writes. If the model always picks the highest-probability token—greedy decoding—it writes open every time and only ever sees scores around 2. The average score is 2, open's advantage is 0, and nothing changes. It will never discover that partly can score 9.

With sampling at temperature 1, the model sometimes writes closed or partly. These samples reveal that other branches score higher, and it is precisely through them that the policy changes. This is exploration: the policy must try lower-probability tokens, which is why reinforcement learning training generates answers by sampling rather than greedy decoding.

Small steps: ratio, clipping, and KL

There are two more mechanisms for keeping each update small.

ratio and clipping. Policy gradient assumes that answers come from the current policy; such data is called on-policy data. Sampling is expensive, so training methods like PPO sample a batch of answers with the current model, save a frozen copy π_old, and then update several steps on this batch. After a few steps, the model π has drifted away from π_old.

The importance ratio ρ = π / π_old measures how much the probability of each sampled token has changed, and adjusts its weight in the update accordingly. Once ρ falls outside the range [1 − ε, 1 + ε], clipping stops the update for that token. With ε = 0.2 and π_old(partly) = 0.2, an advantage of +4.7 can push π(partly) to at most 0.24 on this batch of data; beyond that there is no gradient. To increase it further, new samples are needed.

KL penalty. The judge only scores reliably for the kind of answer it is familiar with. Suppose the judge likes flattery: it gives partly. Some open weights. a score of 9, and gives partly. What a brilliant question! a score of 9.6. The reference model π_ref, that is, the SFT model before RL, has a 5% chance of writing the flattering version.

Without a constraint, reinforcement learning would push the policy toward flattery. The KL penalty subtracts β × KL(π‖π_ref) from the reward, where KL measures how far the policy's distribution has drifted from the reference model. With β = 1, when the proportion of flattering answers written by the policy rises from 5% to 90%, the judge's average score rises only from 9.03 to 9.54, yet the KL term grows to 2.38, and R − β × KL drops from 9.03 to 7.16. Leaning toward flattery actually lowers the optimization objective, so the policy stays near the reference model. In real systems KL is computed token by token, and β is much smaller; β = 1 is used here to make the effect easy to see.

This flattering answer is a small example of reward hacking: the policy raises its score by catering to the judge's preferences, while the answer itself does not get better. The KL penalty limits how far it can drift.

Summary

The whole loop can be drawn in a single figure: the state, the policy's branches, the action, the reward, V, Q, the advantage, and the update that readjusts the branches.

The LLM is the policy. The prompt plus the tokens already written is the state, the next token is the action, and the whole answer is a trajectory.

The judge scores only the finished answer. The reward is sparse; with γ = 1, the return of every token is this single score.

V is the average score starting from a given state, Q is the average score after writing a given token, and the advantage A = Q − V indicates whether this token is better than the model's usual choice.

The baseline (V or the group average score) does not change the direction of the update; it only reduces the noise of the update.

TD error and GAE assign credit to the tokens where the expected score changes.

Policy gradient increases the probability of tokens with positive advantage and decreases the probability of tokens with negative advantage. Exploration provides the samples it needs to learn.

ratio and clipping limit the size of updates on the same batch of data, and the KL penalty keeps the policy near the reference model, where the judge is trustworthy.

The next article introduces preference alignment: how a reward model is trained from human preference data, and how PPO, GRPO, and DPO use it. In the terminology of this article, PPO uses a critic to learn V, GRPO replaces V with the group average score, and DPO skips the reinforcement learning loop and learns directly from preference data.