What Comes After AI Transformers? (Ep. 531)
The Daily AI Show · The Daily AI Show Crew - Brian, Beth, Jyunmi, Andy, Karl, and Eran
Audio is streamed directly from the publisher (anchor.fm) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
The discussion sets the stage for exploring what comes after transformers.
Key Points Discussed
Transformers show limits in reasoning, instruction following, and real-world grounding.
The AI field is moving from scaling to exploring new architectures.
Smarter transformers can be enhanced with test-time compute, neurosymbolic logic, and mixture-of-experts.
Revolutionary alternatives like Mamba, Retinette, and world models introduce different approaches.
Emerging ideas such as spiking neural networks, Kolmogorov Arnold networks, and temporal graph networks may reduce energy costs and improve reasoning.
Neurosymbolic hybrids are highlighted as a promising path for logical reasoning.
The challenge of commercializing research and balancing innovation with environmental costs.
Hybrid futures likely combine multiple architectures into a layered system for AGI.
The concept of swarm intelligence and agent collaboration as another route toward advanced AI.
Timestamps & Topics
00:00:00 💡 Introduction and GPT 5 disappointment
00:02:00 🔍 The shift from scaling to new paradigms
00:04:00 ⚙️ Smarter transformers and test-time compute
00:05:20 🚀 Revolutionary alternatives including Mamba and Retinette
00:06:20 🌍 World models and embodied AI
00:06:58 🧠 Spiking neural networks and novel approaches
00:11:00 ⛵ Exploration analogies and transformer context challenges
00:12:20 🎮 Applications of world models in 3D spaces and XR
00:16:45 🔗 Neurosymbolic hybrids for reasoning
00:19:00 ⚡ Energy efficiency and productization challenges
00:24:00 🌱 Balancing research speed with environmental costs
00:31:00 📉 Four structural limits of transformers
00:35:00 📚 RKV and new memory-efficient mechanisms
00:37:00 📝 Analogies for architectures: note taker, stenographer, librarian, consultant
00:41:00 🕵️ Transformer reasoning illusions and dangers
00:44:00 🔬 Outlier experiments: physical neural nets, temporal graph networks, recurrent GANs
00:49:00 🧩 Hybrid architecture visions for AGI
00:53:30 🐝 Swarm agents and collaborative intelligence
00:55:00 📢 Closing announcements and upcoming shows
The Daily AI Show Co-Hosts: Andy Halliday, Beth Lyons, Brian Maucere, Eran Malloch, Jyunmi Hatcher, and Karl Yeh