Policy Gradients on CartPole
REINFORCE, Actor-Critic, and PPO implemented from scratch in PyTorch: a progression where each method fixes the collapse mode of the one before it.
Problem
Policy-gradient methods all descend from one theorem but fail in characteristic ways. Plain REINFORCE tends to master CartPole and then collapse catastrophically, because its high-variance gradient can shove a good policy off a cliff it cannot climb back from.
How it was solved
Three trainers over one small actor-critic network:
REINFORCE. Episode rollouts and a return-normalized policy gradient. The baseline-free starting point.
Actor-Critic. The value head becomes a learned baseline, so updates weight by advantage rather than raw return. Less variance, still seed-lucky.
PPO. Generalized Advantage Estimation plus a clipped objective: no update moves the policy more than 20% from where it was.
Results
The PPO curve climbs and holds, and its failures are recoverable dips rather than permanent collapses: the signature of the clipped update keeping the policy inside a trust region.
That stability is why PPO is the workhorse in practice, including for RLHF on language models.