Season 2 · Episode 1066
Beyond the Blank Slate: The Evolution of AI Training
Explore the "weight surgery" techniques labs use to expand AI models without losing their core knowledge or starting from zero.
My Weird Prompts · Daniel Rosehill
March 9, 202629m 23s
Audio is streamed directly from the publisher (dts.podtrac.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
Think AI labs start from scratch for every new model? Think again. This episode dives into the high-stakes world of continual pre-training and "weight surgery," where trillion-parameter models are expanded and refined rather than rebuilt at a cost of hundreds of millions. We explore how techniques like Sparse Mixture of Experts and elastic weight consolidation allow models to gain new abilities—like multimodal reasoning—without suffering from catastrophic forgetting. Join us as we pull back the curtain on the biological-style evolution of modern AI and why the "clean slate" is now a relic of the past.