GRPO Explained: The Reinforcement Learning Behind Reasoning Models, with Python Code

When DeepSeek-R1 appeared in January 2025, the surprise was not only how well it reasoned. It was how it learned to reason. The team skipped the usual step of showing the model thousands of worked solutions written by people. They gave it math and coding problems, checked whether each final answer was right, and let reinforcement learning do the rest. The model started to check its own work, go back over its steps and change its approach, with nobody teaching it to. The paper later made the cover of Nature.

The algorithm under the hood was GRPO, Group Relative Policy Optimization. By 2026 it, or a close relative of it, sits inside the training of most open reasoning models, including the Qwen3 and DeepSeek families. Along the way it picked up a set of known flaws, and a family of fixes with names like DAPO, Dr. GRPO, GSPO and CISPO.

This post explains GRPO from the ground up: what problem it solves, how it works, the two hidden biases researchers found in it, and how the 2026 versions fix them. At the end, a short Python example shows one of those biases happen in front of you. If reinforcement learning is new to you, our introduction to reinforcement learning and the Markov decision process explainer cover the basics.

GRPO at a glance

Full name	Group Relative Policy Optimization
Introduced in	DeepSeekMath, by Zhihong Shao and others at DeepSeek, February 2024 (arXiv:2402.03300)
Made famous by	DeepSeek-R1, January 2025, published in Nature in 2025
What it replaces	PPO, the algorithm behind ChatGPT style fine tuning
Main idea	Sample a group of answers to the same question and score each one against the group average, instead of training a second network to predict how good an answer should be
Used in	DeepSeek models, the Qwen3 family (through its GSPO variant), and the GRPO trainer in Hugging Face TRL

Why reinforcement learning for reasoning

A language model is trained in stages. First it reads a huge amount of text and learns to predict the next word. Then it is fine tuned on examples of good answers. For years the last stage was RLHF, reinforcement learning from human feedback: people rank pairs of answers, a reward model learns their taste, and an algorithm called PPO pushes the language model toward answers the reward model likes.

Reasoning tasks offer something better than taste. A math problem has a right answer. A coding problem has tests that pass or fail. You can check the result with a short program, so the reward is exact and cheap. AI2 named this setup RLVR, reinforcement learning with verifiable rewards, in its Tulu 3 work in late 2024.

With an exact reward, the training loop is simple to describe. Give the model a problem, let it write a solution, check the final answer, and make the model more likely to write solutions like the ones that scored well. The hard part is that last step, and that is where GRPO comes in.

From PPO to GRPO

The problem with PPO

PPO judges each answer by its advantage: how much better it did than expected. To know what to expect, PPO trains a second network called a critic, or value model, that looks at the question and the answer so far and predicts the final reward.

For a robot arm, the critic is a small network. For a language model, the critic is usually another language model of about the same size. You now have to hold two big models in memory, train both, and keep the critic accurate while the main model keeps changing. On long reasoning answers, the critic has to guess the final reward from every partial sentence, which it does badly.

The GRPO idea

GRPO drops the critic. For each question it samples a group of answers, often 8 or 16, scores each one with the reward check, and uses the group itself as the baseline.

1. Take a question and sample a group of G answers from the current model.
2. Score every answer. With verifiable rewards this is often 1 for a right final answer and 0 for a wrong one.
3. For each answer, take its reward, subtract the group's average reward, and divide by the group's standard deviation. This is the answer's advantage.
4. Give every token in that answer the same advantage, and update the model with a PPO style clipped loss so no single step changes it too much.

An answer that beats its group gets pushed up. An answer that falls short gets pushed down. The group average plays the role the critic used to play, and it costs nothing to compute. The DeepSeekMath paper presents it as a variant of PPO that improves math reasoning while using less memory.

The original GRPO loss has one more ingredient: a KL penalty that keeps the model close to a frozen copy of itself. Later work found it often unnecessary. DAPO drops it, and Hugging Face TRL sets its weight to zero by default.

The two hidden biases in GRPO

In March 2025, Zichen Liu and colleagues at Sea AI Lab and the National University of Singapore published "Understanding R1-Zero-Like Training: A Critical Perspective". They looked closely at two small normalization steps in the GRPO loss and found that each one quietly bends what the model learns.

Bias 1: wrong answers get longer

In the original GRPO loss, each answer's token losses are averaged over the length of that answer. That sounds harmless. Look at what it does to the size of each update:

• A short right answer spreads its reward over few tokens, so each token gets a big push up. The model learns to keep right answers short.
• A long wrong answer spreads its penalty over many tokens, so each token gets only a small push down. Long wrong answers are punished less than short wrong ones.

The result is a model that drifts toward longer and longer wrong answers. Some of the growth in response length that people celebrated as "the model learning to think longer" may partly be this effect.

Bias 2: easy and hard questions count more

Dividing by the group's standard deviation causes the second bias. When almost every answer in a group is right, or almost every one is wrong, the standard deviation is small, and dividing by a small number makes the advantages large. The paper puts it this way: questions that are too easy or too hard get higher weights during policy updates.

Here is what that looks like for a group of 8 answers with a reward of 1 or 0:

Right answers out of 8	Standard deviation	Scale applied by GRPO
0 or 8	0	no update at all
1 or 7	0.33	3.02
2 or 6	0.43	2.31
4	0.50	2.00

A question the model solves 7 times out of 8 is weighted about 1.5 times more than a question it solves half the time, while the half and half question is where the model has the most to learn.

The first row matters too. When every answer in a group gets the same reward, every advantage is zero and the whole group teaches the model nothing. On a mostly solved dataset, a large share of compute can go to waste this way.

The fix: Dr. GRPO

The paper's fix, Dr. GRPO ("GRPO Done Right"), removes both normalizations. It divides by a fixed constant, such as the maximum answer length, instead of each answer's own length, and it stops dividing by the standard deviation. With this recipe the authors trained Qwen2.5-Math-7B to 43.3% on AIME 2024 in about 27 hours on 8 A100 GPUs.

The GRPO family in 2026

GRPO is now less a single algorithm than a family. These are the variants you will meet most often:

Variant	Who	What it changes
DAPO	ByteDance Seed and Tsinghua AIR, March 2025	Four changes: a higher upper clip limit to keep the model exploring (Clip-Higher), dropping groups where every answer scored the same, averaging the loss over all tokens in the batch instead of per answer, and a softer penalty for answers cut off at the length limit. No KL term. Reached 50 points on AIME 2024 with Qwen2.5-32B, against 47 for DeepSeek-R1-Zero-Qwen-32B, in half the training steps.
Dr. GRPO	Sea AI Lab, March 2025	Removes the per answer length division and the standard deviation division, which removes the two biases above.
GSPO	Qwen team, July 2025	Computes the importance ratio, clipping and optimization over the whole answer instead of token by token. Built to keep training stable on Mixture of Experts models, and used for the Qwen3 models.
CISPO	MiniMax, June 2025	Clips the importance sampling weights instead of clipping token updates, so no token's gradient is thrown away. MiniMax-M1's full reinforcement learning run took three weeks on 512 H800 GPUs, at a rental cost of about $534,700.

These variants mostly share one goal: keep the cheap group baseline, and remove the side effects of the original normalization and clipping. Hugging Face TRL exposes several of them as a single setting in its GRPO trainer, including loss_type="dapo", "dr_grpo" and "cispo".

A small Python example: watching the length bias happen

Bias 1 is easier to believe once you see it. The example below builds a toy "language model" in plain NumPy, with no deep learning library.

For each of 8 questions, the model picks one of 4 answers, then writes filler tokens until it decides to stop, up to 40 tokens. The reward is 1 for the right answer and 0 otherwise, so it is a verifiable reward. Length has nothing to do with being right. Any change in length is purely a side effect of the training method.

We train it twice, once with the original GRPO loss and once with Dr. GRPO, using groups of 8 answers. Dr. GRPO's updates come out smaller, since it divides by the maximum length of 40, so we give it a larger step size to make both learn at a similar pace.

import numpy as np

# A toy "language model". For each of 8 questions it picks one of 4 answers,
# then writes filler tokens until it decides to stop (at most 40 tokens).
# Reward is 1 if the answer is right, 0 otherwise: a verifiable reward.
# questions, answers per question, max answer length, group size
Q, K, MAX_LEN, G = 8, 4, 40, 8
STEPS = 400
# Dr. GRPO divides by MAX_LEN, so its updates are smaller.
# Give it a larger step so both learn at a similar pace.
LR = {"GRPO": 0.5, "Dr. GRPO": 8.0}


def sigmoid(x):
    return 1 / (1 + np.exp(-x))


def expected_length(stop_logits):
    """Average length the policy would write, from its stop probabilities."""
    ps = sigmoid(stop_logits)
    n = np.arange(MAX_LEN - 1)
    lengths = []
    for p in ps.ravel():
        probs = p * (1 - p) ** n          # stop after n filler tokens
        tail = (1 - p) ** (MAX_LEN - 1)   # never stopped
        lengths.append((probs * (n + 2)).sum() + tail * MAX_LEN)
    return np.array(lengths).reshape(stop_logits.shape)


def train(method, seed=0):
    rng = np.random.default_rng(seed)
    answer_logits = rng.normal(0, 1, (Q, K))
    stop_logits = np.full((Q, K), -2.0)  # about 8 filler tokens at the start
    correct = rng.integers(0, K, Q)
    history = []
    for step in range(STEPS):
        g_ans = np.zeros_like(answer_logits)
        g_stop = np.zeros_like(stop_logits)
        wins = 0
        for q in range(Q):
            p = np.exp(answer_logits[q]) / np.exp(answer_logits[q]).sum()
            group = []
            for _ in range(G):  # sample a group of G answers
                a = rng.choice(K, p=p)
                ps = sigmoid(stop_logits[q, a])
                n_cont = min(rng.geometric(ps) - 1, MAX_LEN - 1)
                stopped = n_cont < MAX_LEN - 1
                # answer token + filler tokens + stop token
                length = 1 + n_cont + int(stopped)
                reward = float(a == correct[q])
                group.append((a, n_cont, stopped, length, reward))
            r = np.array([o[4] for o in group])
            adv = r - r.mean()  # group relative: no critic needed
            if method == "GRPO":
                std = r.std()
                adv = adv / (std + 1e-4)
            for (a, n_cont, stopped, length, rew), A in zip(group, adv):
                # weight of each token in the loss
                w = A / length if method == "GRPO" else A / MAX_LEN
                g_ans[q] += w * (np.eye(K)[a] - p)  # answer token
                ps = sigmoid(stop_logits[q, a])
                # filler tokens (continue) and the stop token
                g_stop[q, a] += w * (-ps * n_cont + (1 - ps) * stopped)
                wins += rew
        answer_logits += LR[method] * g_ans / G
        stop_logits += LR[method] * g_stop / G
        L = expected_length(stop_logits)
        is_right = np.eye(K, dtype=bool)[correct]
        accuracy = wins / (Q * G)
        history.append((accuracy, L[is_right].mean(), L[~is_right].mean()))
    return np.array(history)


for method in ["GRPO", "Dr. GRPO"]:
    # average over 5 random seeds
    runs = np.array([train(method, s) for s in range(5)])
    h = runs.mean(axis=0)
    print(method)
    for step in [0, 49, 99, 199, 399]:
        acc, right_len, wrong_len = h[step]
        print(f"  step {step + 1:>3}: accuracy {acc:5.1%}, "
              f"right answers {right_len:5.1f} tokens, "
              f"wrong answers {wrong_len:5.1f} tokens")

Running it prints (about 10 seconds on a laptop):

GRPO
  step   1: accuracy 19.7%, right answers   9.2 tokens, wrong answers   9.4 tokens
  step  50: accuracy 73.8%, right answers   5.2 tokens, wrong answers  11.5 tokens
  step 100: accuracy 90.0%, right answers   3.7 tokens, wrong answers  13.0 tokens
  step 200: accuracy 96.9%, right answers   3.1 tokens, wrong answers  14.3 tokens
  step 400: accuracy 99.1%, right answers   2.8 tokens, wrong answers  15.4 tokens
Dr. GRPO
  step   1: accuracy 19.7%, right answers   9.3 tokens, wrong answers   9.3 tokens
  step  50: accuracy 67.5%, right answers   9.3 tokens, wrong answers   9.4 tokens
  step 100: accuracy 83.1%, right answers   9.2 tokens, wrong answers   9.4 tokens
  step 200: accuracy 95.0%, right answers   9.1 tokens, wrong answers   9.4 tokens
  step 400: accuracy 99.1%, right answers   9.2 tokens, wrong answers   9.4 tokens

Both methods learn the task equally well: 99.1% accuracy after 400 steps. The difference is in how long the answers get.

With GRPO, right answers shrink from about 9 tokens to 2.8, and wrong answers grow from about 9 tokens to 15.4. The model never got any reward for length. The per answer averaging alone did this, exactly as the Dr. GRPO paper describes. With Dr. GRPO, both stay near 9 tokens for the whole run, since nothing in the loss cares about length.

In a real reasoning model, length is not filler. Longer answers can contain more useful reasoning. That is why this bias is hard to spot: a model that writes longer and longer wrong answers can look like a model that is learning to think harder.

Where GRPO is going

Three trends stand out in 2026.

Rewards are becoming environments. Instead of checking one final answer, labs now run the model inside a sandbox with tools, let it act over many steps, and score the result at the end. Hugging Face's August 2026 review of open model reports describes this agentic reinforcement learning in models such as Kimi K3, GLM-5 and MiniMax's agents. The group baseline still works here: sample several attempts at the same task and compare them.

Rewards are becoming hybrids. Not every task has a checkable answer. Recent models combine rule based checks with learned reward models and LLM judges, while keeping the verifiable rewards as an anchor.

Reinforcement learning, then distillation. Several labs now train separate specialist models with GRPO, one per domain, and then distil them into a single model. The same review reports this for DeepSeek-V4.

The same idea shows up in robotics. Our post on reinforcement learning beyond imitation covers robot policies that score their own attempts against an expected outcome and shift toward the better ones. The robot versions mostly train a value function, since a robot cannot cheaply retry the same moment 16 times. For a language model, sampling another answer is cheap, which is what makes the group baseline practical.

Frequently asked questions

Closing thoughts

GRPO succeeded since it made a hard thing cheap: you can judge an answer well enough by comparing it with its siblings, without training a second model to do the judging. Its flaws were small details in the loss, but small details in a loss shape everything the model learns. That is the most useful lesson in this story. When a model starts writing longer answers, check the loss before you celebrate.

If you run the example, try lowering MAX_LEN or raising the group size, and watch how the gap between right and wrong answers changes. We would like to hear what you find.

References

• Z. Shao, P. Wang, Q. Zhu and others, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arXiv:2402.03300
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. Nature 645, 633 to 638. arXiv:2501.12948
• Q. Yu and others, DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025. arXiv:2503.14476
• Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee and M. Lin, Understanding R1-Zero-Like Training: A Critical Perspective, 2025. arXiv:2503.20783
• C. Zheng and others, Group Sequence Policy Optimization, Qwen team, 2025. arXiv:2507.18071
• MiniMax, MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention, 2025. arXiv:2506.13585
• Hugging Face TRL, GRPO Trainer documentation. huggingface.co/docs/trl
• S. Paniego, How frontier models train on outcomes in 2026, Hugging Face blog, August 2026. huggingface.co/blog