Doom Debates!

Martin Casado is a General Partner at Andreessen Horowitz (a16z) who has strong views about AI.He claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.I set out to understand his worldview around AI, and pinpoint the crux of disagreement with my own view.Spoiler: I conclude that Martin needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.00:00 Introducing Martin Casado01:42 Martin’s AGI Timeline05:39 Martin’s Analysis of Self-Driving Cars15:30 Heavy-Tail Distributions38:03 Understanding General Intelligence38:29 AI's Progress in Specific Domains43:20 AI’s Understanding of Meaning47:16 Compression and Intelligence48:09 Symbol Grounding53:24 Human Abstractions and AI01:18:18 The Frontier of AI Applications01:23:04 Human vs. AI: Concept Creation and Reasoning01:25:51 The Complexity of the Universe and AI's Limitations01:28:16 AI's Potential in Biology and Simulation01:32:40 The Essence of Intelligence and Creativity in AI01:41:13 AI's Future Capabilities02:00:29 Intelligence vs. Simulation02:14:59 AI Regulation02:23:05 Concluding ThoughtsWatch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz.Follow Martin: @martin_casadoFollow Nate: @labenzFollow Liron: @lironSubscribe to the Doom Debates YouTube Channel to get full episodes plus other bonus content!Search “Doom Debates” to subscribe in your podcast player. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Tilek Mamutov is a Kyrgyzstani software engineer who worked at Google X for 11 years before founding his own international software engineer recruiting company, Outtalent.Since first encountering the AI doom argument at a Center for Applied Rationality bootcamp 10 years ago, he considers it a serious possibility, but he doesn’t currently feel convinced that doom is likely.Let’s explore Tilek’s worldview and pinpoint where he gets off the doom train and why!00:12 Tilek’s Background01:43 Life in Kyrgyzstan04:32 Tilek’s Non-Doomer Position07:12 Debating AI Doom Scenarios13:49 Nuclear Weapons and AI Analogies39:22 Privacy and Empathy in Human-AI Interaction39:43 AI's Potential in Understanding Human Emotions41:14 The Debate on AI's Empathy Capabilities42:23 Quantum Effects and AI's Predictive Models45:33 The Complexity of AI Control and Safety47:10 Optimization Power: AI vs. Human Intelligence48:39 The Risks of AI Self-Replication and Control51:52 Historical Analogies and AI Safety Concerns56:35 The Challenge of Embedding Safety in AI Goals01:02:42 The Future of AI: Control, Optimization, and Risks01:15:54 The Fragility of Security Systems01:16:56 Debating AI Optimization and Catastrophic Risks01:18:34 The Outcome Pump Thought Experiment01:19:46 Human Persuasion vs. AI Control01:21:37 The Crux of Disagreement: Robustness of AI Goals01:28:57 Slow vs. Fast AI Takeoff Scenarios01:38:54 The Importance of AI Alignment01:43:05 ConclusionFollow Tilekx.com/tilekLinksI referenced Paul Christiano’s scenario of gradual AI doom, a slower version that doesn’t require a Yudkowskian “foom”. Worth a read: What Failure Looks LikeI also referenced the concept of “edge instantiation” to explain that if you’re optimizing powerfully for some metric, you don’t get other intuitively nice things as a bonus, you *just* get the exact thing your function is measuring. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Mike Israetel is a well-known bodybuilder and fitness influencer with over 600,000 Instagram followers, and a surprisingly intelligent commentator on other subjects, including a whole recent episode on the AI alignment problem:Mike brought up many interesting points that were worth responding to, making for an interesting reaction episode. I also appreciate that he’s helping get the urgent topic of AI alignment in front of a mainstream audience.Unfortunately, Mike doesn’t engage with the possibility that AI alignment is an intractable technical problem on a 5-20 year timeframe, which I think is more likely than not. That’s the crux of why he and I disagree, and why I see most of his episode as talking past most other intelligent positions people take on AI alignment. I hope he’ll keep engaging with the topic and rethink his position.00:00 Introduction03:08 AI Risks and Scenarios06:42 Superintelligence Arms Race12:39 The Importance of AI Alignment18:10 Challenges in Defining Human Values26:11 The Outer and Inner Alignment Problems44:00 Transhumanism and AI's Potential45:42 The Next Step In Evolution47:54 AI Alignment and Potential Catastrophes50:48 Scenarios of AI Development54:03 The AI Alignment Problem01:07:39 AI as a Helper System01:08:53 Corporations and AI Development01:10:19 The Risk of Unaligned AI01:27:18 Building a Superintelligent AI01:30:57 ConclusionFollow Mike Israetel:* instagram.com/drmikeisraetel* youtube.com/@MikeIsraetelMakingProgressGet the full Doom Debates experience:* Subscribe to youtube.com/@DoomDebates* Subscribe to this Substack: DoomDebates.com* Search "Doom Debates" to subscribe in your podcast player* Follow me at x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

What did we learn from my debate with Robin Hanson? Did we successfully isolate the cruxes of disagreement? I actually think we did!In this post-debate analysis, we’ll review what those key cruxes are, and why I still think I’m right and Robin is wrong about them!I’ve taken the time to think much harder about everything Robin said during the debate, so I can give you new & better counterarguments than the ones I was able to make in realtime.Timestamps00:00 Debate Reactions06:08 AI Timelines and Key Metrics08:30 “Optimization Power” vs. “Innovation”11:49 Economic Growth and Diffusion17:56 Predicting Future Trends24:23 Crux of Disagreement with Robin’s Methodology34:59 Conjunction Argument for Low P(Doom)37:26 Headroom Above Human Intelligence41:13 The Role of Culture in Human Intelligence48:01 Goal-Completeness and AI Optimization50:48 Misaligned Foom Scenario59:29 Monitoring AI and the Rule of Law01:04:51 How Robin Sees Alignment01:09:08 Reflecting on the DebateLinksAISafety.info - The fractal of counterarguments to non-doomers’ argumentsFor the full Doom Debates experience:* Subscribe to youtube.com/@DoomDebates* Subscribe to this Substack: DoomDebates.com* Search "Doom Debates" to subscribe in your podcast player* Follow me at x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Robin Hanson is a legend in the rationality community and one of my biggest intellectual influences.In 2008, he famously debated Eliezer Yudkowsky about AI doom via a sequence of dueling blog posts known as the great Hanson-Yudkowsky Foom Debate. This debate picks up where Hanson-Yudkowsky left off, revisiting key arguments in the light of recent AI advances.My position is similar to Eliezer's: P(doom) is on the order of 50%. Robin's position is shockingly different: P(doom) is below 1%.00:00 Announcements03:18 Debate Begins05:41 Discussing AI Timelines and Predictions19:54 Economic Growth and AI Impact31:40 Outside Views vs. Inside Views on AI46:22 Predicting Future Economic Growth51:10 Historical Doubling Times and Future Projections54:11 Human Brain Size and Economic Metrics57:20 The Next Era of Innovation01:07:41 AI and Future Predictions01:14:24 The Vulnerable World Hypothesis01:16:27 AI Foom01:28:15 Genetics and Human Brain Evolution01:29:24 The Role of Culture in Human Intelligence01:31:36 Brain Size and Intelligence Debate01:33:44 AI and Goal-Completeness01:35:10 AI Optimization and Economic Impact01:41:50 Feasibility of AI Alignment01:55:21 AI Liability and Regulation02:05:26 Final Thoughts and Wrap-UpRobin's links:Twitter: x.com/RobinHansonHome Page: hanson.gmu.eduRobin’s top related essays:* What Are Reasonable AI Fears?* AIs Will Be Our Mind ChildrenPauseAI links:https://pauseai.info/https://discord.gg/2XXWXvErfACheck out https://youtube.com/@ForHumanityPodcast, the other podcast raising the alarm about AI extinction!For the full Doom Debates experience:* Subscribe to https://youtube.com/@DoomDebates* Subscribe to the Substack: https://DoomDebates.com* Search "Doom Debates" to subscribe in your podcast player* Follow me at https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

This episode is a comprehensive preparation session for my upcoming debate on AI doom with the legendary Robin Hanson.Robin’s P(doom) is I’ve researched past debates, blogs, tweets, and scholarly discussions related to AI doom, and plan to focus our debate on the cruxes of disagreement between Robin’s position and my own Eliezer Yudkowsky-like position.Key topics include the probability of humanity’s extinction due to uncontrollable AGI, alignment strategies, AI capabilities and timelines, the impact of AI advancements, and various predictions made by Hanson.00:00 Introduction03:37 Opening Statement04:29 Value-Extinction Spectrum05:34 Future AI Capabilities08:23 AI Timelines13:23 What can't current AIs do15:48 Architecture/Algorithms vs. Content17:40 Cyc18:55 Is intelligence many different things, or one thing?19:31 Goal-Completeness20:44 AIXI22:10 Convergence in AI systems23:02 Foom26:00 Outside view: Extrapolating robust trends26:18 Salient Events Timeline30:56 Eliezer's claim about meta-levels affecting capability growth rates33:53 My claim - the optimization power model trumps these outside-view trends35:19 Aren't there many other possible outside views?37:03 Is alignment feasible?40:14 What's the warning shot that would make you concerned?41:07 Future Foom evidence?44:59 How else have Robin's views changed in the last decade?Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack — DoomDebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos — youtube.com/@doomdebates4. Follow me on Twitter — x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

I’ve been studying Robin Hanson’s catalog of writings and interviews in preparation for our upcoming AI doom debate. Now I’m doing an exercise where I step into Robin’s shoes, and make the strongest possible case for his non-doom position!This exercise is called the Ideological Turing Test, and it’s based on the idea that it’s only productive to argue against someone if you understand what you’re arguing against. Being able to argue *for* a position proves that you understand it.My guest David Xu is a fellow AI doomer, and deep thinker, who volunteered to argue the doomer position against my version of non-doomer “Robin”.00:00 Upcoming Debate with Dr. Robin Hanson01:15 David Xu's Background and Perspective02:23 The Ideological Turing Test02:39 David's AI Doom Claim03:44 AI Takeover vs. Non-AI Descendants05:21 Paperclip Maximizer15:53 Economic Trends and AI Predictions27:18 Recursive Self-Improvement and Foom29:14 Comparing Models of Intelligence34:53 The Foom Scenario36:04 Coordination and Lawlessness in AI37:49 AI's Goal-Directed Behavior and Economic Models40:02 Multipolar Outcomes and AI Coordination40:58 The Orthogonality Thesis and AI Firms43:18 AI's Potential to Exceed Human Control45:03 The Argument for AI Misalignment48:22 Economic Trends vs. AI Catastrophes59:13 The Race for AI Dominance01:04:09 AI Escaping Control01:04:45 AI Liability and Insurance01:06:14 Economic Dynamics and AI Threats01:07:18 The Balance of Offense and Defense in AI01:08:38 AI's Potential to Disrupt National Infrastructure01:10:17 The Multipolar Outcome of AI Development01:11:00 Human Role in AI-Driven Future01:12:19 Debating the Discontinuity in AI Progress01:25:26 Closing Statements and Final Thoughts01:30:34 Reflecting on the Debate and Future DiscussionsFollow David: https://x.com/davidxu90The Ideological Turing Test (ITT) was coined by Bryan Caplan in this classic post: https://www.econlib.org/archives/2011/06/the_ideological.htmlI also did a Twitter version of the ITT here: https://x.com/liron/status/1789688119773872273Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@doomdebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I'm answering questions from listener Tony Warren.1:16 Biological imperatives in machine learning2:22 Evolutionary pressure vs. AI training4:15 Instrumental convergence and AI goals6:46 Human vs. AI problem domains9:20 AI vs. human actuators18:04 Evolution and intelligence33:23 Maximum intelligence54:55 Computational limits and the futureFollow Tony: https://x.com/Pove_iOS---Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@doomdebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

My guest Rob thinks superintelligent AI will suffer from analysis paralysis from trying to achieve a 100% probability of killing humanity. Since AI won’t be satisfied with 99.9% of defeating us, it won’t dare to try, and we’ll live!Doom Debates catalogues all the different stops where people get off the “doom train”, all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.Follow Rob: https://x.com/LoB_BlacksageIf you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@DoomDebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m debating the one & only Professor Steven Pinker!!! Well, I kind of am, in my head. Let me know if you like this format…Dr. Pinker is optimistic that AI doom worries are overblown. But I find his arguments shallow, and I’m disappointed with his overall approach to the AI doom discourse.Here’s the full video of Steven Pinker talking to Michael C. Moynihan on this week’s episode of “Honestly with Bari Weiss”: https://youtube.com/watch?v=mTuH1Ucbif4If you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@DoomDebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

RJ, a pseudonymous listener, volunteered to debate me.Follow RJ: https://x.com/impershblknightIf you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@doomdebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Danny asks:> You've said that an intelligent AI would lead to doom because it would be an excellent goal-to-action mapper. A great football coach like Andy Reid is a great goal-to-action mapper. He's on the sidelines, but he knows exactly what actions his team needs to execute to achieve the goal and win the game. > But if he had a team of chimpanzees or elementary schoolers, or just players who did not want to cooperate, then his team would not execute his plans and they would lose. And even his very talented team of highly motivated players who also want to win the game, sometimes execute his actions badly. Now an intelligent AI that does not control a robot army has very limited ability to perform precise acts in the physical world. From within the virtual world, an AI would not be able to get animals or plants to carry out specific actions that it wants performed. I don't see how the AI could get monkeys or dolphins to maintain power plants or build chips.> The AI needs humans to carry out its plans, but in the real physical world, when dealing with humans, knowing what you want people to do is a small part of the equation. Won't the AI in practice struggle to get humans to execute its plans in the precise way that it needs?Follow Danny: https://x.com/Danno28_Follow Liron: https://x.com/lironPlease join my email list: DoomDebates.com Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m going to play you my debate with the brilliant hacker and entrepreneur, George Hotz.This took place on an X Space last August.Prior to our debate, George had done a debate with Eliezer Yudkowsky on Dwarkesh Podcast:Follow George: https://x.com/realGeorgeHotzFollow Liron: https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Chase Mann claims accelerating AGI timelines is the best thing we can do for the survival of the 8 billion people alive today.I claim pausing AI is still the highest-expected-utility decision for everyone.Who do you agree with? Comment on my Substack/X/YouTube and let me know!Follow Chase:https://x.com/ChaseMannFollow Liron:https://x.com/lironLessWrong has some great posts about cryonics: https://www.lesswrong.com/tag/cryonics Get full access to Doom Debates at lironshapira.substack.com/subscribe

It’s a monologue episode!Robin Hanson’s blog: https://OvercomingBias.comRobin Hanson’s famous concept, the Great Filter: https://en.wikipedia.org/wiki/Great_FilterRobin Hanson’s groundbreaking 2021 solution to the Fermi Paradox: https://GrabbyAliens.comRobin Hanson’s conversation with Ronny Fernandez about AI doom from May 2023: My tweet about whether we can hope to control superintelligent AI by judging its explanations and arguments: https://x.com/liron/status/1798135026166698239Zvi Mowshowitz’s blog where he posts EXCELLENT weekly AI roundups: https://thezvi.wordpress.comA takedown of Chris Dixon (Andreessen Horowitz)’s book about the nonsensical “Web3” pitch, which despite being terribly argued, is able to trick a significant number of readers into thinking they just read a good argument: https://www.citationneeded.news/review-read-write-own-by-chris-dixon/(Or maybe you think Chris’s book makes total sense, in which case you can observe that a significant number of smart people somehow don’t get how much sense it makes.)Eliezer Yudkowsky’s famous post about Newcomb’s Problem: https://www.lesswrong.com/posts/6ddcsdA2c2XpNpE5x/newcomb-s-problem-and-regret-of-rationality Get full access to Doom Debates at lironshapira.substack.com/subscribe

Welcome and thanks for listening!* Why is Liron finally starting a podcast?* Who does Liron want to debate?* What’s the debate format?* What are Liron’s credentials?* Is someone “rational” like Liron actually just a religious cult member?Follow Ori on Twitter: https://x.com/ygrowthcoMake sure to subscribe for more episodes! Get full access to Doom Debates at lironshapira.substack.com/subscribe

Kelvin is optimistic that the forces of economic competition will keep AIs sufficiently aligned with humanity by the time they become superintelligent.He thinks AIs and humans will plausibly use interoperable money systems (powered by crypto).So even if our values diverge, the AIs will still uphold a system that respects ownership rights, such that humans may hold onto a nontrivial share of capital with which to pursue human values.I view these kinds of scenarios as wishful thinking with probability much lower than that of the simple undignified scenario I expect, wherein the first uncontrollable AGI correctly realizes what dodos we are in both senses of the word. Get full access to Doom Debates at lironshapira.substack.com/subscribe