Reinforcement learning

Overview

Reinforcement learning (RL) is the branch of machine learning where the training signal is a reward that arrives after a sequence of decisions, not a label attached to each input. The agent chooses actions, the environment responds with a new state and a scalar reward, and the agent must learn a policy that maximises the sum of future rewards. That makes RL the right tool when a decision changes what happens next (robot control, scheduling, game playing, dialogue) and the wrong tool when a labelled dataset already exists.

For a working engineer the subject matters for two reasons at the time of writing. First, the same algorithm, proximal policy optimisation (PPO), and its relatives are what turned pretrained language models into assistants, and the post-training recipes in /topics/fine-tuning-and-alignment/ are RL whether or not they say so. Second, RL fails in ways that supervised learning does not: silently, by exploiting a reward you wrote carelessly.

Key ideas

The Markov decision process is the contract

An MDP has states, actions, a transition function, a reward function and a discount factor. The policy maps states to a distribution over actions. The value function estimates expected discounted return from a state, and the Q function does the same for a state-action pair. Everything in RL is a way of estimating one of these from samples. Value-based methods such as DQN (Mnih et al., 2015) learn Q and act greedily. Policy-gradient methods learn the policy directly by nudging the probability of actions that led to higher return. Actor-critic methods, which include PPO, learn both and use the value estimate to reduce the variance of the policy gradient.

The discount factor and the reward are design decisions, not facts about the world. A reward of plus one for reaching the goal and zero otherwise is sparse and slow to learn from. Shaping it with intermediate rewards speeds learning and invites the agent to collect the intermediate rewards without ever reaching the goal.

PPO: a policy gradient with a seat belt

A naive policy-gradient update can take a step so large that the new policy is worse everywhere and the collected data no longer describes it. Trust-region methods constrain the step. PPO (Schulman et al., 2017) replaces the constraint with a clipped objective: the ratio of new to old action probability is clipped to a band such as 0.8 to 1.2, so an advantage cannot be chased past the band. Advantages come from generalised advantage estimation, which blends one-step and Monte Carlo returns with a parameter lambda. The loop below is the one in every PPO implementation, including the language-model ones.

Offline RL: learning from a fixed dataset

Online RL assumes you can collect more data from the current policy. In medicine, industrial control or anything with a real cost per mistake, you have a log of past decisions and nothing else. Running an off-policy algorithm on that log fails, because the Q function is asked about actions the log never contains and extrapolates optimistically. Offline methods such as conservative Q-learning (Kumar et al., 2020) penalise values for unseen actions, and sequence-model approaches such as the Decision Transformer treat RL as conditional sequence prediction. The honest finding from Levine et al.’s survey is that offline RL beats behaviour cloning when the data is diverse and the reward is informative, and loses to it when the log already contains expert behaviour.

RL for language models

Reinforcement learning from human feedback (Ouyang et al., 2022) trains a reward model on human preferences between pairs of responses, then runs PPO with the language model as the policy, each token as an action, and the reward model score at the end of the sequence as the reward. A KL penalty against the frozen starting model stops the policy from drifting into gibberish that the reward model happens to like. Direct preference optimisation (Rafailov et al., 2023) removes the reward model and the rollout loop by rewriting the objective as a classification loss on the preference pairs, which is cheaper but cannot use rewards from a verifier. Group relative policy optimisation (Shao et al., 2024) keeps the online loop but drops the value network: it samples a group of responses per prompt and uses the group mean as the baseline. DeepSeek-R1 (671 B total, 37 B active, 128 k context, MIT licence, January 2025) used GRPO with verifiable rewards on maths and code, and the maker reports that chain-of-thought reasoning emerged from the reward alone rather than from demonstrations.

Where to start

Work through /learn/beginner/04-training-loop-and-loss/ and /learn/beginner/07-your-first-neural-network/ so that a policy network and a value network are ordinary objects to you. Then read /learn/beginner/09-using-an-llm-well/ to see what the post-trained model looks like from the outside. For hands-on practice, implement REINFORCE on CartPole in Gymnasium before touching a library, then compare your result with Stable-Baselines3’s PPO. The /reference/pytorch-training-loop/ sheet covers the loop you will be modifying.

Tools and platforms

Common mistakes

• Using RL where supervised learning fits. If you have labels, or can get them, train on them. RL is the fallback for sequential decisions with delayed reward.
• Trusting a single seed. RL results vary enormously across seeds. Report at least five and plot the spread.
• Writing a reward and not watching the agent. Reward hacking is the default outcome, not an edge case. Record videos or transcripts of the policy early and often.
• Forgetting the KL term when post-training a language model, or setting it so high that nothing changes. Sweep it.
• Running offline RL on data from a single expert policy and expecting improvement over behaviour cloning.
• Skipping observation and reward normalisation. PPO on raw rewards of scale 1,000 will not learn.

Going deeper

• Sutton and Barto, Reinforcement Learning: An Introduction, second edition, 2018, free online.
• Schulman et al., Proximal Policy Optimization Algorithms, 2017.
• Ouyang et al., Training language models to follow instructions with human feedback, 2022.
• Rafailov et al., Direct Preference Optimization, 2023.
• Shao et al., DeepSeekMath, 2024, the paper that introduced GRPO.
• Levine et al., Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020.