GRPO (Group Relative Policy Optimisation) is a reinforcement learning algorithm designed specifically for training language models from reward signals, introduced by DeepSeek in 2024 as a more compute-efficient alternative to PPO (Proximal Policy Optimisation). Standard PPO requires a separate critic (value) network to estimate baselines, roughly doubling the memory and compute cost. GRPO eliminates the critic by generating a group of candidate responses for each prompt, scoring them with a reward model, and using the mean reward of the group as a dynamic baseline — hence "group relative." The policy is then updated to increase the probability of responses that scored above the group average and decrease the probability of those that scored below.

Think of it like a teacher who gives the same essay prompt to a class of students, then grades all their papers and awards extra credit only to students who scored above the class average — and penalises those below it. The class average becomes the benchmark, so no separate "ideal answer" reference is needed.

A simplified GRPO update for a group of G completions per prompt looks like this:

rewards = reward_model(completions)          # shape: [G]
baseline = rewards.mean()                    # group average
advantages = rewards - baseline              # relative scores
# Policy gradient step: increase prob of high-advantage completions
loss = -(advantages * log_probs).mean()
loss.backward()
optimizer.step()

GRPO connects closely to RLHF (the broader paradigm of aligning models with human preferences via RL), Loss Function (GRPO defines a policy-gradient objective over reward advantages), Inference (generating the G candidate responses is the main computational cost), and Foundation Model alignment workflows where reinforcement fine-tuning follows supervised pretraining. DeepSeek-R1 used GRPO to develop its chain-of-thought reasoning capabilities with notably lower GPU cost than PPO-based approaches.

You will encounter GRPO when reading alignment research papers, evaluating open-source RL fine-tuning libraries (TRL, OpenRLHF), or setting up a reasoning-focused fine-tuning pipeline on a budget that cannot support two simultaneous large model instances.