A reward model is a neural network trained to predict how much a human would prefer one AI output over another, producing a scalar score that substitutes for direct human judgment during RLHF training. It is built by fine-tuning a pretrained language model on a dataset of pairwise human preferences-annotators compare two responses to the same prompt and mark the better one-so the reward model internalises what "good" looks like.

Think of it like training a wine-quality judge. You cannot have a master sommelier taste every bottle a winery produces, so you collect their judgements on thousands of samples, train a model to replicate their taste, and then use that model to score new batches at scale. The reward model is the automated judge.

In a typical RLHF pipeline: (1) a Foundation Model generates many response pairs; (2) human raters label which response is better; (3) a reward model is trained on those labels; (4) reinforcement learning-usually PPO-updates the language model's weights to maximise the reward model's scores. The quality of the final assistant is heavily determined by the reward model's calibration and coverage.

The reward model is central to RLHF and directly connects to Reinforcement Learning, Fine-tuning, and Loss Function design; its imperfections are the primary driver of reward hacking, where the policy learns to game the proxy score rather than genuinely improve. Constitutional AI and process-reward models are emerging alternatives that attempt to address its limitations.

You will encounter reward models in technical discussions of ChatGPT, Claude, Gemini, and any other RLHF-aligned assistant, and in alignment research exploring how to make AI systems reliably helpful and safe.