GRPO Algorithm Reference
Overview
Group Relative Policy Optimization (GRPO) is a policy gradient method that eliminates the need for a learned critic by computing advantages from group statistics. For each prompt, multiple completions are sampled and their rewards are normalized within the group to produce advantages.
Key insight: Instead of training a value function to estimate V(s), GRPO uses the mean reward of the group as the baseline. This removes the critic entirely, reducing memory and avoiding value function approximation errors.
Training Loop
Each step follows this pipeline:
1. Sample G completions per prompt from current policy 2. Decode completions to text 3. Score completions with reward function 4. Compute group-relative advantages 5. Compute per-token log-probs (current policy + reference policy) 6. Compute clipped surrogate loss with KL penalty 7. Backpropagate and update
Advantage Estimation
Rewards are normalized within each group of G completions for the same prompt:
mu = mean(r_1, ..., r_G) sigma = std(r_1, ..., r_G) A_i = (r_i - mu) / (sigma + epsilon)
epsilon is a small constant (typically 1e-4 to 1e-8) for numerical stability only. It prevents division by zero when all rewards in a group are identical.
Properties:
- Advantages within each group sum to approximately zero;
- High-reward completions get positive advantages, low-reward get negative;
- The learning signal vanishes if epsilon is too large (dominates denominator) or if rewards are constant. For example, if all rewards in a group are identical (or nearly identical), then causing all advantages to collapse to 0;
Log-Probability Computation
Per-token log probabilities via log-softmax:
log_prob(token_i) = logit(token_i) - logsumexp(logits)
Critical invariant: log_prob <= 0 always, since it's the log of a probability in (0, 1].
Sequence-level log probability: log_pi(y|x) = sum_t log_pi(t_j | x, t_<j)
Loss Function
The GRPO loss combines a clipped surrogate objective with a KL divergence penalty:
ratio = exp(log_pi_theta(y|x) - log_pi_old(y|x)) L_clip = min(ratio * A, clip(ratio, 1-eps, 1+eps) * A) L_kl = beta * KL(pi_theta || pi_ref) loss = -E[L_clip] + L_kl
| Component | Purpose |
|---|---|
ratio | How much the policy has changed from the generation policy |
| Clipping | Prevents destructively large policy updates |
beta * KL | Keeps the policy close to the reference (prevents degeneration) |
Key Hyperparameters
| Parameter | Typical Range | Effect |
|---|---|---|
num_generations (G) | 4–16 | More = lower variance advantages, higher compute cost |
beta | 0.01–0.1 | Higher = more conservative updates (closer to reference) |
epsilon (clip) | 0.1–0.2 | Narrower = more conservative updates |
epsilon (advantage) | 1e-8–1e-4 | Must be small; only for numerical stability |
learning_rate | 1e-7–5e-6 | Much lower than SFT; RL is sensitive to LR |
Available References
| File | Contents | When to load |
|---|---|---|
references/grpo-algorithm.md | Full mathematical formulation with notation table, step-by-step derivations, comparison to PPO and REINFORCE | When you need to verify whether a specific implementation detail matches the algorithm specification |
references/grpo-trainer-internals.md | GRPOTrainer implementation from popular frameworks (TRL): method-by-method breakdown, data flow diagram, and how each algorithm step maps to code | When tracing bugs through a GRPOTrainer implementation or understanding how the algorithm maps to specific code paths |
