RL and RLHF

Plain English. Reinforcement learning (RL) trains a model by reward and penalty over many attempts rather than by showing it correct answers. RLHF — reinforcement learning from human feedback — is the variant where the reward comes from humans rating outputs. RLHF is what made chatbots polite and useful in 2022; RL against harder, checkable objectives is what makes reasoning models capable now.

Why it moves money. These are different cost structures. RLHF runs on a human-labour supply chain — annotation firms, preference raters — which is a real, investable (and displaceable) market. Outcome-based RL runs on compute and environments instead, and labs are shifting spend towards it because machine-checkable rewards scale where human raters don't. The distinction matters: a lab saying "we scaled RL" is claiming capability; "we scaled RLHF" is mostly claiming behaviour.

What to watch. The share of training compute going to RL, disclosed or estimated, and whether annotation vendors' revenue growth survives the shift to verifiable and model-generated rewards.

From the signals. A single-rollout async RL paper claimed stability gains over GRPO — the algorithm layer is still moving. Dan Luu asked why labs haven't built RL environments for testing.

Further reading. Christiano et al., Deep RL from Human Preferences (2017) · Ouyang et al., the InstructGPT paper (2022)