“How Well Does RL Scale?” by Toby_Ord

<p> This is the latest in a series of essays on AI Scaling. <br> You can find the others on my site.</p><p> Summary: RL-training for LLMs scales surprisingly poorly. Most of its gains are from allowing LLMs to productively use longer chains of thought, allowing them to think longer about a problem. There is some improvement for a fixed length of answer, but not enough to drive AI progress. Given the scaling up of pre-training compute also stalled, we'll see less AI progress via compute scaling than you might have thought, and more of it will come from inference scaling (which has different effects on the world). That lengthens timelines and affects strategies for AI governance and safety.</p><p> </p><p> The current era of improving AI capabilities using reinforcement learning (from verifiable rewards) involves two key types of scaling:</p><ol> <li> Scaling the amount of compute used for RL during training</li><li> Scaling [...]</li></ol></br></p> <p>---</p><p><strong>Outline:</strong></p><p>(09:12) How do these compare to pre-training scaling?</p><p>(13:42) Conclusion</p> <p>---</p> <p><b>First published:</b><br/> October 22nd, 2025 </p> <p><b>Source:</b><br/> <a href="https://forum.effectivealtruism.org/posts/TysuCdgwDnQjH3LyY/how-well-does-rl-scale?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://forum.effectivealtruism.org/posts/TysuCdgwDnQjH3LyY/how-well-does-rl-scale</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=ea_forum&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p> <p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/rroqzdlnwd9umtbaabhh" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/rroqzdlnwd9umtbaabhh" alt="Two scatter plots showing AIME accuracy during training and testing periods. Both charts demonstrate performance improvement as computational resources increase." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/zfrfdactzimwnz7cxcxd" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/zfrfdactzimwnz7cxcxd" alt="Bar graph titled "Ludicrous rate of progress" comparing Grok versions' compute efficiency." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/ysc36ugs4kswlaiitadw" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/ysc36ugs4kswlaiitadw" alt="Graph showing AIME 2025 performance versus estimated inference costs for different models." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/vrp1lk4xxpygjqfzokgu" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/vrp1lk4xxpygjqfzokgu" alt="Two graphs showing "o1 AIME accuracy" during training and testing times." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/kj4wkrnq0ytsnsd7evis" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/kj4wkrnq0ytsnsd7evis" alt="Graph showing Math level 5 performance with base model and RL post-training." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/i4jhzhkkqagke7i0tt9b" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/i4jhzhkkqagke7i0tt9b" alt="Graph comparing GPT-5 and OpenAI o3 software engineering accuracy versus token output." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/fckvphzzqs3yzcxltifl" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/fckvphzzqs3yzcxltifl" alt="Graph showing "AIME 2022/2023 performance during training" comparing OpenAI models' computational efficiency." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/s2ixldlwlfauc9rhrygv" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/s2ixldlwlfauc9rhrygv" alt=""ARC-AGI-1 Leaderboard" graph showing AI model performance versus cost per task." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/wjb9n2sd4ty97op9adj8" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/wjb9n2sd4ty97op9adj8" alt="Graph comparing GPT-5 and OpenAI o3 performance on PhD-level science questions." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/yqlxnpkw1dkxqjcuvr7i" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/TysuCdgwDnQjH3LyY/yqlxnpkw1dkxqjcuvr7i" alt="Graph showing performance vs tokens for OTIS Mock AIME 2024-25 models." style="max-width: 100%;" /></a><p><em>Apple Podcasts and Spotify do not show images in the episode description. Try <a href="https://pocketcasts.com/" target="_blank" rel="noreferrer">Pocket Casts</a>, or another podcast app.</em></p></div>