Proximal Policy Optimisation (PPO) is a reinforcement learning algorithm that trains an agent's decision-making policy by iteratively collecting experience and updating the policy with gradient ascent, while deliberately limiting how far each update can move from the previous policy to prevent destabilising the training. The "proximal" constraint is enforced via a clipped surrogate objective that discards gradient steps that would cause the new policy to deviate too far - measured by the probability ratio between old and new policies - ensuring stable, monotonic improvement. PPO has become the dominant algorithm for RLHF alignment pipelines, where a reward model trained on human preferences guides the policy updates that shape LLM behaviour.

Think of it like learning to ride a bicycle by making small, deliberate adjustments to your balance after each wobble, rather than overcorrecting so violently that you fall off in the opposite direction - steady incremental updates beat dramatic overhauls.

A simplified PPO update loop pseudocode:

for iteration in range(num_iterations):
    rollouts = collect_trajectories(policy_old, env)   # sample actions
    rewards = compute_returns(rollouts)

    for epoch in range(ppo_epochs):
        ratio = policy_new.prob(actions) / policy_old.prob(actions)
        clipped = torch.clamp(ratio, 1 - eps, 1 + eps)  # proximal constraint
        loss = -torch.min(ratio * advantages, clipped * advantages).mean()
        optimiser.zero_grad(); loss.backward(); optimiser.step()

    policy_old.load_state_dict(policy_new.state_dict())

PPO's role in RLHF connects it to Fine-tuning (PPO fine-tunes the base LLM using reward signals) and the Loss Function (the clipped objective is PPO's custom loss). It is also a key Hyperparameter-sensitive algorithm - clip range, learning rate, and KL penalty all require careful tuning. Alternatives like DPO avoid the RL loop entirely but PPO remains the reference approach for complex reward shaping.

You will encounter PPO when reading about how models like ChatGPT or Claude are aligned: after supervised fine-tuning, a PPO-based RLHF stage adjusts the model toward human-preferred responses, making PPO central to modern conversational AI safety and capability work.