Future of Life Institute Podcast

Maryanna Saenko joins the podcast to discuss how venture capital works, how to fund innovation, and what the fields of investing and philanthropy could learn from each other. You can read more about Maryanna's work at https://future.ventures Timestamps: 00:00 How does venture capital work? 09:01 Failure and success for startups 13:22 Is overconfidence necessary? 19:20 Repeat entrepreneurs 24:38 Long-term investing 30:36 Feedback loops from investments 35:05 Timing investments 38:35 The hardware-software dichotomy 42:19 Innovation prizes 45:43 VC lessons for philanthropy 51:03 Creating new markets 54:01 Investing versus philanthropy 56:14 Technology preying on human frailty 1:00:55 Are good ideas getting harder to find? 1:06:17 Artificial intelligence 1:12:41 Funding ethics research 1:14:25 Is philosophy useful? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Connor Leahy joins the podcast to discuss the state of the AI. Which labs are in front? Which alignment solutions might work? How will the public react to more capable AI? You can read more about Connor's work at https://conjecture.dev Timestamps: 00:00 Landscape of AI research labs 10:13 Is AGI a useful term? 13:31 AI predictions 17:56 Reinforcement learning from human feedback 29:53 Mechanistic interpretability 33:37 Yudkowsky and Christiano 41:39 Cognitive Emulations 43:11 Public reactions to AI Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Connor Leahy joins the podcast to discuss GPT-4, magic, cognitive emulation, demand for human-like AI, and aligning superintelligence. You can read more about Connor's work at https://conjecture.dev Timestamps: 00:00 GPT-4 16:35 "Magic" in machine learning 27:43 Cognitive emulations 38:00 Machine learning VS explainability 48:00 Human data = human AI? 1:00:07 Analogies for cognitive emulations 1:26:03 Demand for human-like AI 1:31:50 Aligning superintelligence Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Lennart Heim joins the podcast to discuss options for governing the compute used by AI labs and potential problems with this approach to AI safety. You can read more about Lennart's work here: https://heim.xyz/about/ Timestamps: 00:00 Introduction 00:37 AI risk 03:33 Why focus on compute? 11:27 Monitoring compute 20:30 Restricting compute 26:54 Subsidising compute 34:00 Compute as a bottleneck 38:41 US and China 42:14 Unintended consequences 46:50 Will AI be like nuclear energy? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Lennart Heim joins the podcast to discuss how we can forecast AI progress by researching AI hardware. You can read more about Lennart's work here: https://heim.xyz/about/ Timestamps: 00:00 Introduction 01:00 The AI triad 06:26 Modern chip production 15:54 Forecasting AI with compute 27:18 Running out of data? 32:37 Three eras of AI training 37:58 Next chip paradigm 44:21 AI takeoff speeds Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Liv Boeree joins the podcast to discuss poker, GPT-4, human-AI interaction, whether this is the most important century, and building a dataset of human wisdom. You can read more about Liv's work here: https://livboeree.com Timestamps: 00:00 Introduction 00:36 AI in Poker 09:35 Game-playing AI 13:45 GPT-4 and generative AI 26:41 Human-AI interaction 32:05 AI arms race risks 39:32 Most important century? 42:36 Diminishing returns to intelligence? 49:14 Dataset of human wisdom/meaning Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Liv Boeree joins the podcast to discuss Moloch, beauty filters, game theory, institutional change, and artificial intelligence. You can read more about Liv's work here: https://livboeree.com Timestamps: 00:00 Introduction 01:57 What is Moloch? 04:13 Beauty filters 10:06 Science citations 15:18 Resisting Moloch 20:51 New institutions 26:02 Moloch and WinWin 28:41 Changing systems 33:37 Artificial intelligence 39:14 AI acceleration Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Tobias Baumann joins the podcast to discuss suffering risks, space colonization, and cooperative artificial intelligence. You can read more about Tobias' work here: https://centerforreducingsuffering.org. Timestamps: 00:00 Suffering risks 02:50 Space colonization 10:12 Moral circle expansion 19:14 Cooperative artificial intelligence 36:19 Influencing governments 39:34 Can we reduce suffering? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Tobias Baumann joins the podcast to discuss suffering risks, artificial sentience, and the problem of knowing which actions reduce suffering in the long-term future. You can read more about Tobias' work here: https://centerforreducingsuffering.org. Timestamps: 00:00 Introduction 00:52 What are suffering risks? 05:40 Artificial sentience 17:18 Is reducing suffering hopelessly difficult? 26:06 Can we know how to reduce suffering? 31:17 Why are suffering risks neglected? 37:31 How do we avoid accidentally increasing suffering? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Neel Nanda joins the podcast for a lightning round on mathematics, technological progress, aging, living up to our values, and generative AI. You can find his blog here: https://www.neelnanda.io Timestamps: 00:00 Introduction 00:55 How useful is advanced mathematics? 02:24 Will AI replace mathematicians? 03:28 What are the key drivers of tech progress? 04:13 What scientific discovery would disrupt Neel's worldview? 05:59 How should humanity view aging? 08:03 How can we live up to our values? 10:56 What can we learn from a person who lived 1.000 years ago? 12:05 What should we do after we have aligned AGI? 16:19 What important concept is often misunderstood? 17:22 What is the most impressive scientific discovery? 18:08 Are language models better learning tools that textbooks? 21:22 Should settling Mars be a priority for humanity? 22:44 How can we focus on our work? 24:04 Are human-AI relationships morally okay? 25:18 Are there aliens in the universe? 26:02 What are Neel's favourite books? 27:15 What is an overlooked positive aspect of humanity? 28:33 Should people spend more time prepping for disaster? 30:41 Neel's advice for teens. 31:55 How will generative AI evolve over the next five years? 32:56 How much can AIs achieve through a web browser? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Neel Nanda joins the podcast to talk about mechanistic interpretability and how it can make AI safer. Neel is an independent AI safety researcher. You can find his blog here: https://www.neelnanda.io Timestamps: 00:00 Introduction 00:46 How early is the field mechanistic interpretability? 03:12 Why should we care about mechanistic interpretability? 06:38 What are some successes in mechanistic interpretability? 16:29 How promising is mechanistic interpretability? 31:13 Is machine learning analogous to evolution? 32:58 How does mechanistic interpretability make AI safer? 36:54 36:54 Does mechanistic interpretability help us control AI? 39:57 Will AI models resist interpretation? 43:43 Is mechanistic interpretability fast enough? 54:10 Does mechanistic interpretability give us a general understanding? 57:44 How can you help with mechanistic interpretability? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Neel Nanda joins the podcast to explain how we can understand neural networks using mechanistic interpretability. Neel is an independent AI safety researcher. You can find his blog here: https://www.neelnanda.io Timestamps: 00:00 Who is Neel? 04:41 How did Neel choose to work on AI safety? 12:57 What does an AI safety researcher do? 15:53 How analogous are digital neural networks to brains? 21:34 Are neural networks like alien beings? 29:13 Can humans think like AIs? 35:00 Can AIs help us discover new physics? 39:56 How advanced is the field of AI safety? 45:56 How did Neel form independent opinions on AI? 48:20 How does AI safety research decrease the risk of extinction? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Connor Leahy from Conjecture joins the podcast for a lightning round on a variety of topics ranging from aliens to education. Learn more about Connor's work at https://conjecture.dev Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Connor Leahy from Conjecture joins the podcast to discuss AI safety, the fragility of the world, slowing down AI development, regulating AI, and the optimal funding model for AI safety research. Learn more about Connor's work at https://conjecture.dev Timestamps: 00:00 Introduction 00:47 What is the best way to understand AI safety? 09:50 Why is the world relatively stable? 15:18 Is the main worry human misuse of AI? 22:47 Can humanity solve AI safety? 30:06 Can we slow down AI development? 37:13 How should governments regulate AI? 41:09 How do we avoid misallocating AI safety government grants? 51:02 Should AI safety research be done by for-profit companies? Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Connor Leahy from Conjecture joins the podcast to discuss AI progress, chimps, memes, and markets. Learn more about Connor's work at https://conjecture.dev Timestamps: 00:00 Introduction 01:00 Defining artificial general intelligence 04:52 What makes humans more powerful than chimps? 17:23 Would AIs have to be social to be intelligent? 20:29 Importing humanity's memes into AIs 23:07 How do we measure progress in AI? 42:39 Gut feelings about AI progress 47:29 Connor's predictions about AGI 52:44 Is predicting AGI soon betting against the market? 57:43 How accurate are prediction markets about AGI?

On this special episode of the podcast, Emilia Javorsky interviews Sean Ekins about regulating AI drug discovery. Timestramps: 00:00 Introduction 00:31 Ethical guidelines and regulation of AI drug discovery 06:11 How do we balance innovation and safety in AI drug discovery? 13:12 Keeping dangerous chemical data safe 21:16 Sean’s personal story of voicing concerns about AI drug discovery 32:06 How Sean will continue working on AI drug discovery

On this special episode of the podcast, Emilia Javorsky interviews Sean Ekins about the dangers of AI drug discovery. They talk about how Sean discovered an extremely toxic chemical (VX) by reversing an AI drug discovery algorithm. Timestamps: 00:00 Introduction 00:46 Sean’s professional journey 03:45 Can computational models replace animal models? 07:24 The risks of AI drug discovery 12:48 Should scientists disclose dangerous discoveries? 19:40 How should scientists handle dual-use technologies? 22:08 Should we open-source potentially dangerous discoveries? 26:20 How do we control autonomous drug creation? 31:36 Surprising chemical discoveries made by black-box AI systems 36:56 How could the dangers of AI drug discovery be mitigated?

Anders Sandberg joins the podcast to discuss various philosophical questions about the value of the future. Learn more about Anders' work: https://www.fhi.ox.ac.uk Timestamps: 00:00 Introduction 00:54 Humanity as an immature teenager 04:24 How should we respond to our values changing over time? 18:53 How quickly should we change our values? 24:58 Are there limits to what future morality could become? 29:45 Could the universe contain infinite value? 36:00 How do we balance weird philosophy with common sense? 41:36 Lightning round: mind uploading, aliens, interstellar travel, cryonics

Anders Sandberg joins the podcast to discuss how big the future could be and what humanity could achieve at the limits of physics. Learn more about Anders' work: https://www.fhi.ox.ac.uk Timestamps: 00:00 Introduction 00:58 Does it make sense to write long books now? 06:53 Is it possible to understand all of science now? 10:44 What is exploratory engineering? 15:48 Will humanity develop a completed science? 21:18 How much of possible technology has humanity already invented? 25:22 Which sciences have made the most progress? 29:11 How materially wealthy could humanity become? 39:34 Does a grand futures depend on space travel? 49:16 Trade between proponents of different moral theories 53:13 How does physics limit our ethical options? 55:24 How much could our understanding of physics change? 1:02:30 The next episode

Anders Sandberg from The Future of Humanity Institute joins the podcast to discuss ChatGPT, large language models, and what he's learned about the risks and benefits of AI. Timestamps: 00:00 Introduction 00:40 ChatGPT 06:33 Will AI continue to surprise us? 16:22 How do language models fail? 24:23 Language models trained on their own output 27:29 Can language models write college-level essays? 35:03 Do language models understand anything? 39:59 How will AI models improve in the future? 43:26 AI safety in light of recent AI progress 51:28 AIs should be uncertain about values

Vincent Boulanin joins the podcast to explain how modern militaries use AI, including in nuclear weapons systems. Learn more about Vincent's work: https://sipri.org Timestamps: 00:00 Introduction 00:45 Categorizing risks from AI and nuclear 07:40 AI being used by non-state actors 12:57 Combining AI with nuclear technology 15:13 A human should remain in the loop 25:05 Automation bias 29:58 Information requirements for nuclear launch decisions 35:22 Vincent's general conclusion about military machine learning 37:22 Specific policy measures for decreasing nuclear risk Social Media Links: ➡️ WEBSITE: https://futureoflife.org ➡️ TWITTER: https://twitter.com/FLIxrisk ➡️ INSTAGRAM: https://www.instagram.com/futureoflifeinstitute/ ➡️ META: https://www.facebook.com/futureoflifeinstitute ➡️ LINKEDIN: https://www.linkedin.com/company/future-of-life-institute/

Vincent Boulanin joins the podcast to explain the dangers of incorporating artificial intelligence in nuclear weapons systems. Learn more about Vincent's work: https://sipri.org Timestamps: 00:00 Introduction 00:55 What is strategic stability? 02:45 How can AI be a positive factor in nuclear risk? 10:17 Remote sensing of nuclear submarines 19:50 Using AI in nuclear command and control 24:21 How does AI change the game theory of nuclear war? 30:49 How could AI cause an accidental nuclear escalation? 36:57 How could AI cause an inadvertent nuclear escalation? 43:08 What is the most important problem in AI nuclear risk? 44:39 The next episode

Robin Hanson joins the podcast to discuss AI forecasting methods and metrics. Timestamps: 00:00 Introduction 00:49 Robin's experience working with AI 06:04 Robin's views on AI development 10:41 Should we care about metrics for AI progress? 16:56 Is it useful to track AI progress? 22:02 When should we begin worrying about AI safety? 29:16 The history of AI development 39:52 AI progress that deviates from current trends 43:34 Is this AI boom different than past booms? 48:26 Different metrics for predicting AI

Robin Hanson joins the podcast to explain his theory of grabby aliens and its implications for the future of humanity. Learn more about the theory here: https://grabbyaliens.com Timestamps: 00:00 Introduction 00:49 Why should we care about aliens? 05:58 Loud alien civilizations and quiet alien civilizations 08:16 Why would some alien civilizations be quiet? 14:50 The moving parts of the grabby aliens model 23:57 Why is humanity early in the universe? 28:46 Could't we just be alone in the universe? 33:15 When will humanity expand into space? 46:05 Will humanity be more advanced than the aliens we meet? 49:32 What if we discovered aliens tomorrow? 53:44 Should the way we think about aliens change our actions? 57:48 Can we reasonably theorize about aliens? 53:39 The next episode

Ajeya Cotra joins us to talk about thinking clearly in a rapidly changing world. Learn more about the work of Ajeya and her colleagues: https://www.openphilanthropy.org Timestamps: 00:00 Introduction 00:44 The default versus the accelerating picture of the future 04:25 The role of AI in accelerating change 06:48 Extrapolating economic growth 08:53 How do we know whether the pace of change is accelerating? 15:07 How can we cope with a rapidly changing world? 18:50 How could the future be utopian? 22:03 Is accelerating technological progress immoral? 25:43 Should we imagine concrete future scenarios? 31:15 How should we act in an accelerating world? 34:41 How Ajeya could be wrong about the future 41:41 What if change accelerates very rapidly?

Ajeya Cotra joins us to discuss how artificial intelligence could cause catastrophe. Follow the work of Ajeya and her colleagues: https://www.openphilanthropy.org Timestamps: 00:00 Introduction 00:53 AI safety research in general 02:04 Realistic scenarios for AI catastrophes 06:51 A dangerous AI model developed in the near future 09:10 Assumptions behind dangerous AI development 14:45 Can AIs learn long-term planning? 18:09 Can AIs understand human psychology? 22:32 Training an AI model with naive safety features 24:06 Can AIs be deceptive? 31:07 What happens after deploying an unsafe AI system? 44:03 What can we do to prevent an AI catastrophe? 53:58 The next episode

Ajeya Cotra joins us to discuss forecasting transformative artificial intelligence. Follow the work of Ajeya and her colleagues: https://www.openphilanthropy.org Timestamps: 00:00 Introduction 00:53 Ajeya's report on AI 01:16 What is transformative AI? 02:09 Forecasting transformative AI 02:53 Historical growth rates 05:10 Simpler forecasting methods 09:01 Biological anchors 16:31 Different paths to transformative AI 17:55 Which year will we get transformative AI? 25:54 Expert opinion on transformative AI 30:08 Are today's machine learning techniques enough? 33:06 Will AI be limited by the physical world and regulation? 38:15 Will AI be limited by training data? 41:48 Are there human abilities that AIs cannot learn? 47:22 The next episode

Alan Robock joins us to discuss nuclear winter, famine and geoengineering. Learn more about Alan's work: http://people.envsci.rutgers.edu/robock/ Follow Alan on Twitter: https://twitter.com/AlanRobock Timestamps: 00:00 Introduction 00:45 What is nuclear winter? 06:27 A nuclear war between India and Pakistan 09:16 Targets in a nuclear war 11:08 Why does the world have so many nuclear weapons? 19:28 Societal collapse in a nuclear winter 22:45 Should we prepare for a nuclear winter? 28:13 Skepticism about nuclear winter 35:16 Unanswered questions about nuclear winter

Brian Toon joins us to discuss the risk of nuclear winter. Learn more about Brian's work: https://lasp.colorado.edu/home/people/brian-toon/ Read Brian's publications: https://airbornescience.nasa.gov/person/Brian_Toon Timestamps: 00:00 Introduction 01:02 Asteroid impacts 04:20 The discovery of nuclear winter 13:56 Comparing volcanoes and asteroids to nuclear weapons 19:42 How did life survive the asteroid impact 65 million years ago? 25:05 How humanity could go extinct 29:46 Nuclear weapons as a great filter 34:32 Nuclear winter and food production 40:58 The psychology of nuclear threat 43:56 Geoengineering to prevent nuclear winter 46:49 Will humanity avoid nuclear winter?

Philip Reiner joins us to talk about nuclear, command, control and communications systems. Learn more about Philip’s work: https://securityandtechnology.org/ Timestamps: [00:00:00] Introduction [00:00:50] Nuclear command, control, and communications [00:03:52] Old technology in nuclear systems [00:12:18] Incentives for nuclear states [00:15:04] Selectively enhancing security [00:17:34] Unilateral de-escalation [00:18:04] Nuclear communications [00:24:08] The CATALINK System [00:31:25] AI in nuclear command, control, and communications [00:40:27] Russia's war in Ukraine

Daniela and Dario Amodei join us to discuss Anthropic: a new AI safety and research company that's working to build reliable, interpretable, and steerable AI systems. Topics discussed in this episode include: -Anthropic's mission and research strategy -Recent research and papers by Anthropic -Anthropic's structure as a "public benefit corporation" -Career opportunities You can find the page for the podcast here: https://futureoflife.org/2022/03/04/daniela-and-dario-amodei-on-anthropic/ Watch the video version of this episode here: https://www.youtube.com/watch?v=uAA6PZkek4A Careers at Anthropic: https://www.anthropic.com/#careers Anthropic's Transformer Circuits research: https://transformer-circuits.pub/ Follow Anthropic on Twitter: https://twitter.com/AnthropicAI microCOVID Project: https://www.microcovid.org/ Follow Lucas on Twitter: https://twitter.com/lucasfmperry Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:44 What was the intention behind forming Anthropic? 6:28 Do the founders of Anthropic share a similar view on AI? 7:55 What is Anthropic's focused research bet? 11:10 Does AI existential safety fit into Anthropic's work and thinking? 14:14 Examples of AI models today that have properties relevant to future AI existential safety 16:12 Why work on large scale models? 20:02 What does it mean for a model to lie? 22:44 Safety concerns around the open-endedness of large models 29:01 How does safety work fit into race dynamics to more and more powerful AI? 36:16 Anthropic's mission and how it fits into AI alignment 38:40 Why explore large models for AI safety and scaling to more intelligent systems? 43:24 Is Anthropic's research strategy a form of prosaic alignment? 46:22 Anthropic's recent research and papers 49:52 How difficult is it to interpret current AI models? 52:40 Anthropic's research on alignment and societal impact 55:35 Why did you decide to release tools and videos alongside your interpretability research 1:01:04 What is it like working with your sibling? 1:05:33 Inspiration around creating Anthropic 1:12:40 Is there an upward bound on capability gains from scaling current models? 1:18:00 Why is it unlikely that continuously increasing the number of parameters on models will lead to AGI? 1:21:10 Bootstrapping models 1:22:26 How does Anthropic see itself as positioned in the AI safety space? 1:25:35 What does being a public benefit corporation mean for Anthropic? 1:30:55 Anthropic's perspective on windfall profits from powerful AI systems 1:34:07 Issues with current AI systems and their relationship with long-term safety concerns 1:39:30 Anthropic's plan to communicate it's work to technical researchers and policy makers 1:41:28 AI evaluations and monitoring 1:42:50 AI governance 1:45:12 Careers at Anthropic 1:48:30 What it's like working at Anthropic 1:52:48 Why hire people of a wide variety of technical backgrounds? 1:54:33 What's a future you're excited about or hopeful for? 1:59:42 Where to find and follow Anthropic This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Anthony Aguirre and Anna Yelizarova join us to discuss FLI's new Worldbuilding Contest. Topics discussed in this episode include: -Motivations behind the contest -The importance of worldbuilding -The rules of the contest -What a submission consists of -Due date and prizes Learn more about the contest here: https://worldbuild.ai/ Join the discord: https://discord.com/invite/njZyTJpwMz You can find the page for the podcast here: https://futureoflife.org/2022/02/08/anthony-aguirre-and-anna-yelizarova-on-flis-worldbuilding-contest/ Watch the video version of this episode here: https://www.youtube.com/watch?v=WZBXSiyienI Follow Lucas on Twitter here: twitter.com/lucasfmperry Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:30 What is "worldbuilding" and FLI's Worldbuilding Contest? 6:32 Why do worldbuilding for 2045? 7:22 Why is it important to practice worldbuilding? 13:50 What are the rules of the contest? 19:53 What does a submission consist of? 22:16 Due dates and prizes? 25:58 Final thoughts and how the contest contributes to creating beneficial futures This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

David Chalmers, Professor of Philosophy and Neural Science at NYU, joins us to discuss his newest book Reality+: Virtual Worlds and the Problems of Philosophy. Topics discussed in this episode include: -Virtual reality as genuine reality -Why VR is compatible with the good life -Why we can never know whether we're in a simulation -Consciousness in virtual realities -The ethics of simulated beings You can find the page for the podcast here: https://futureoflife.org/2022/01/26/david-chalmers-on-reality-virtual-worlds-and-the-problems-of-philosophy/ Watch the video version of this episode here: https://www.youtube.com/watch?v=hePEg_h90KI Check out David's book and website here: http://consc.net/ Follow Lucas on Twitter here: https://twitter.com/lucasfmperry Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:43 How this books fits into David's philosophical journey 9:40 David's favorite part(s) of the book 12:04 What is the thesis of the book? 14:00 The core areas of philosophy and how they fit into Reality+ 16:48 Techno-philosophy 19:38 What is "virtual reality?" 21:06 Why is virtual reality "genuine reality?" 25:27 What is the dust theory and what's it have to do with the simulation hypothesis? 29:59 How does the dust theory fit in with arguing for virtual reality as genuine reality? 34:45 Exploring criteria for what it means for something to be real 42:38 What is the common sense view of what is real? 46:19 Is your book intended to address common sense intuitions about virtual reality? 48:51 Nozick's experience machine and how questions of value fit in 54:20 Technological implementations of virtual reality 58:40 How does consciousness fit into all of this? 1:00:18 Substrate independence and if classical computers can be conscious 1:02:35 How do problems of identity fit into virtual reality? 1:04:54 How would David upload himself? 1:08:00 How does the mind body problem fit into Reality+? 1:11:40 Is consciousness the foundation of value? 1:14:23 Does your moral theory affect whether you can live a good life in a virtual reality? 1:17:20 What does a good life in virtual reality look like? 1:19:08 David's favorite VR experiences 1:20:42 What is the moral status of simulated people? 1:22:38 Will there be unconscious simulated people with moral patiency? 1:24:41 Why we can never know we're not in a simulation 1:27:56 David's credences for whether we live in a simulation 1:30:29 Digital physics and what is says about the simulation hypothesis 1:35:21 Imperfect realism and how David sees the world after writing Reality+ 1:37:51 David's thoughts on God 1:39:42 Moral realism or anti-realism? 1:40:55 Where to follow David and find Reality+ This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Rohin Shah, Research Scientist on DeepMind's technical AGI safety team, joins us to discuss: AI value alignment; how an AI Researcher might decide whether to work on AI Safety; and why we don't know that AI systems won't lead to existential risk. Topics discussed in this episode include: - Inner Alignment versus Outer Alignment - Foundation Models - Structural AI Risks - Unipolar versus Multipolar Scenarios - The Most Important Thing That Impacts the Future of Life You can find the page for the podcast here: https://futureoflife.org/2021/11/01/rohin-shah-on-the-state-of-agi-safety-research-in-2021 Watch the video version of this episode here: https://youtu.be/_5xkh-Rh6Ec Follow the Alignment Newsletter here: https://rohinshah.com/alignment-newsletter/ Have any feedback about the podcast? You can share your thoughts here: https://www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 00:02:22 What is AI alignment? 00:06:00 How has your perspective of this problem changed over the past year? 00:06:28 Inner Alignment 00:13:00 Ways that AI could actually lead to human extinction 00:18:53 Inner Alignment and MACE optimizers 00:20:15 Outer Alignment 00:23:12 The core problem of AI alignment 00:24:54 Learning Systems versus Planning Systems 00:28:10 AI and Existential Risk 00:32:05 The probability of AI existential risk 00:51:31 Core problems in AI alignment 00:54:46 How has AI alignment, as a field of research changed in the last year? 00:54:02 Large scale language models 00:54:50 Foundation Models 00:59:58 Why don't we know that AI systems won't totally kill us all? 01:09:05 How much of the alignment and safety problems in AI will be solved by industry? 01:14:44 Do you think about what beneficial futures look like? 01:19:31 Moral Anti-Realism and AI 01:27:25 Unipolar versus Multipolar Scenarios 01:35:33 What is the safety team at DeepMind up to? 01:35:41 What is the most important thing that impacts the future of life? This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Future of Life Institute President Max Tegmark and our grants team, Andrea Berman and Daniel Filan, join us to announce a $25M multi-year AI Existential Safety Grants Program. Topics discussed in this episode include: - The reason Future of Life Institute is offering AI Existential Safety Grants - Max speaks about how receiving a grant changed his career early on - Daniel and Andrea provide details on the fellowships and future grant priorities Check out our grants programs here: https://grants.futureoflife.org/ Join our AI Existential Safety Community: https://futureoflife.org/team/ai-exis... Have any feedback about the podcast? You can share your thoughts here: https://www.surveymonkey.com/r/DRBFZCT This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Dr. Filippa Lentzos, Senior Lecturer in Science and International Security at King's College London, joins us to discuss the most pressing issues in biosecurity, big data in biology and life sciences, and governance in biological risk. Topics discussed in this episode include: - The most pressing issue in biosecurity - Stories from when biosafety labs failed to contain dangerous pathogens - The lethality of pathogens being worked on at biolaboratories - Lessons from COVID-19 You can find the page for the podcast here: https://futureoflife.org/2021/10/01/filippa-lentzos-on-emerging-threats-in-biosecurity/ Watch the video version of this episode here: https://www.youtube.com/watch?v=I6M34oQ4v4w Have any feedback about the podcast? You can share your thoughts here: https://www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:35 What are the least understood aspects of biological risk? 8:32 Which groups are interested biotechnologies that could be used for harm? 16:30 Why countries may pursue the development of dangerous pathogens 18:45 Dr. Lentzos' strands of research 25:41 Stories from when biosafety labs failed to contain dangerous pathogens 28:34 The most pressing issue in biosecurity 31:06 What is gain of function research? What are the risks? 34:57 Examples of gain of function research 36:14 What are the benefits of gain of function research? 37:54 The lethality of pathogens being worked on at biolaboratorie 40:25 Benefits and risks of big data in biology and the life sciences 45:03 Creating a bioweather map or using big data for biodefense 48:35 Lessons from COVID-19 53:46 How does governance fit in to biological risk? 55:59 Key takeaways from Dr. Lentzos This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Susan Solomon, internationally recognized atmospheric chemist, and Stephen Andersen, leader of the Montreal Protocol, join us to tell the story of the ozone hole and their roles in helping to bring us back from the brink of disaster. Topics discussed in this episode include: -The industrial and commercial uses of chlorofluorocarbons (CFCs) -How we discovered the atmospheric effects of CFCs -The Montreal Protocol and its significance -Dr. Solomon's, Dr. Farman's, and Dr. Andersen's crucial roles in helping to solve the ozone hole crisis -Lessons we can take away for climate change and other global catastrophic risks You can find the page for this podcast here: https://futureoflife.org/2021/09/16/susan-solomon-and-stephen-andersen-on-saving-the-ozone-layer/ Check out the video version of the episode here: https://www.youtube.com/watch?v=7hwh-uDo-6A&ab_channel=FutureofLifeInstitute Check out the story of the ozone hole crisis here: https://undsci.berkeley.edu/article/0_0_0/ozone_depletion_01 Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 3:13 What are CFCs and what was their role in society? 7:09 James Lovelock discovering an abundance of CFCs in the lower atmosphere 12:43 F. Sherwood Rowland's and Mario Molina's research on the atmospheric science of CFCs 19:52 How a single chlorine atom from a CFC molecule can destroy a large amount of ozone 23:12 Moving from models of ozone depletion to empirical evidence of the ozone depleting mechanism 24:41 Joseph Farman and discovering the ozone hole 30:36 Susan Solomon's discovery of the surfaces of high altitude Arctic clouds being crucial for ozone depletion 47:22 The Montreal Protocol 1:00:00 Who were the key stake holders in the Montreal Protocol? 1:03:46 Stephen Andersen's efforts to phase out CFCs as the co-chair of the Montreal Protocol Technology and Economic Assessment Panel 1:13:28 The Montreal Protocol helping to prevent 11 billion metric tons of CO2 emissions per year 1:18:30 Susan and Stephen's key takeaways from their experience with the ozone hole crisis 1:24:24 What world did we avoid through our efforts to save the ozone layer? 1:28:37 The lessons Stephen and Susan take away from their experience working to phase out CFCs from industry 1:34:30 Is action on climate change practical? 1:40:34 Does the Paris Agreement have something like the Montreal Protocol Technology and Economic Assessment Panel? 1:43:23 Final words from Susan and Stephen This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

James Manyika, Chairman and Director of the McKinsey Global Institute, joins us to discuss the rapidly evolving landscape of the modern global economy and the role of technology in it. Topics discussed in this episode include: -The modern social contract -Reskilling, wage stagnation, and inequality -Technology induced unemployment -The structure of the global economy -The geographic concentration of economic growth You can find the page for this podcast here: https://futureoflife.org/2021/09/06/james-manyika-on-global-economic-and-technological-trends/ Check out the video version of the episode here: https://youtu.be/zLXmFiwT0-M Check out the McKinsey Global Institute here: https://www.mckinsey.com/mgi/overview Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:14 What are the most important problems in the world today? 4:30 The issue of inequality 8:17 How the structure of the global economy is changing 10:21 How does the role of incentives fit into global issues? 13:00 How the social contract has evolved in the 21st century 18:20 A billion people lifted out of poverty 19:04 What drives economic growth? 29:28 How does AI automation affect the virtuous and vicious versions of productivity growth? 38:06 Automation and reflecting on jobs lost, jobs gained, and jobs changed 43:15 AGI and automation 48:00 How do we address the issue of technology induced unemployment 58:05 Developing countries and economies 1:01:29 The central forces in the global economy 1:07:36 The global economic center of gravity 1:09:42 Understanding the core impacts of AI 1:12:32 How do global catastrophic and existential risks fit into the modern global economy? 1:17:52 The economics of climate change and AI risk 1:20:50 Will we use AI technology like we've used fossil fuel technology? 1:24:34 The risks of AI contributing to inequality and bias 1:31:45 How do we integrate developing countries voices in the development and deployment of AI systems 1:33:42 James' core takeaway 1:37:19 Where to follow and learn more about James' work This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Michael Klare, Five College Professor of Peace & World Security Studies, joins us to discuss the Pentagon's view of climate change, why it's distinctive, and how this all ultimately relates to the risks of great powers conflict and state collapse. Topics discussed in this episode include: -How the US military views and takes action on climate change -Examples of existing climate related difficulties and what they tell us about the future -Threat multiplication from climate change -The risks of climate change catalyzed nuclear war and major conflict -The melting of the Arctic and the geopolitical situation which arises from that -Messaging on climate change You can find the page for this podcast here: https://futureoflife.org/2021/07/30/michael-klare-on-the-pentagons-view-of-climate-change-and-the-risks-of-state-collapse/ Check out the video version of the episode here: https://www.youtube.com/watch?v=bn57jxEoW24 Check out Michael's website here: http://michaelklare.com/ Apply for the Podcast Producer position here: futureoflife.org/job-postings/ Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:28 How does the Pentagon view climate change and why are they interested in it? 5:30 What are the Pentagon's main priorities besides climate change? 8:31 What are the objectives of career officers at the Pentagon and how do they see climate change? 10:32 The relationship between Pentagon career officers and the Trump administration on climate change 15:47 How is the Pentagon's view of climate change unique and important? 19:54 How climate change exacerbates existing difficulties and the issue of threat multiplication 24:25 How will climate change increase the tensions between the nuclear weapons states of India, Pakistan, and China? 26:32 What happened to Tacloban City and how is it relevant? 32:27 Why does the US military provide global humanitarian assistance? 34:39 How has climate change impacted the conditions in Nigeria and how does this inform the Pentagon's perspective? 39:40 What is the ladder of escalation for climate change related issues? 46:54 What is "all hell breaking loose?" 48:26 What is the geopolitical situation arising from the melting of the Arctic? 52:48 Why does the Bering Strait matter for the Arctic? 54:23 The Arctic as a main source of conflict for the great powers in the coming years 58:01 Are there ongoing proposals for resolving territorial disputes in the Arctic? 1:01:40 Nuclear weapons risk and climate change 1:03:32 How does the Pentagon intend to address climate change? 1:06:20 Hardening US military bases and going green 1:11:50 How climate change will affect critical infrastructure 1:15:47 How do lethal autonomous weapons fit into the risks of escalation in a world stressed by climate change? 1:19:42 How does this all affect existential risk? 1:24:39 Are there timelines for when climate change induced stresses will occur? 1:27:03 Does tying existential risks to national security issues benefit awareness around existential risk? 1:30:18 Does relating climate change to migration issues help with climate messaging? 1:31:08 A summary of the Pentagon's interest, view, and action on climate change 1:33:00 Final words from Michael 1:34:33 Where to find more of Michael's work This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Avi Loeb, Professor of Science at Harvard University, joins us to discuss unidentified aerial phenomena and a recent US Government report assessing their existence and threat. Topics discussed in this episode include: -Evidence counting for the natural, human, and extraterrestrial origins of UAPs -The culture of science and how it deals with UAP reports -How humanity should respond if we discover UAPs are alien in origin -A project for collecting high quality data on UAPs You can find the page for this podcast here: https://futureoflife.org/2021/07/09/avi-loeb-on-ufos-and-if-theyre-alien-in-origin/ Apply for the Podcast Producer position here: futureoflife.org/job-postings/ Check out the video version of the episode here: https://www.youtube.com/watch?v=AyNlLaFTeFI&ab_channel=FutureofLifeInstitute Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 1:41 Why is the US Government report on UAPs significant? 7:08 Multiple different sensors detecting the same phenomena 11:50 Are UAPs a US technology? 13:20 Incentives to deploy powerful technology 15:48 What are the flight and capability characteristics of UAPs? 17:53 The similarities between 'Oumuamua and UAP reports 20:11 Are UAPs some form of spoofing technology? 22:48 What is the most convincing natural or conventional explanation of UAPs? 25:09 UAPs as potentially containing artificial intelligence 28:15 Can you give a credence to UAPs being alien in origin? 29:32 Why aren't UAPs far more technologically advanced? 32:15 How should humanity respond if UAPs are found to be alien in origin? 35:15 A plan to get better data on UAPs 38:56 Final thoughts from Avi 39:40 Getting in contact with Avi to support his project This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Avi Loeb, Professor of Science at Harvard University, joins us to discuss a recent interstellar visitor, if we've already encountered alien technology, and whether we're ultimately alone in the cosmos. Topics discussed in this episode include: -Whether 'Oumuamua is alien or natural in origin -The culture of science and how it affects fruitful inquiry -Looking for signs of alien life throughout the solar system and beyond -Alien artefacts and galactic treaties -How humanity should handle a potential first contact with extraterrestrials -The relationship between what is true and what is good You can find the page for this podcast here: https://futureoflife.org/2021/07/09/avi-loeb-on-oumuamua-aliens-space-archeology-great-filters-and-superstructures/ Apply for the Podcast Producer position here: https://futureoflife.org/job-postings/ Check out the video version of the episode here: https://www.youtube.com/watch?v=qcxJ8QZQkwE&ab_channel=FutureofLifeInstitute See our second interview with Avi here: https://soundcloud.com/futureoflife/avi-loeb-on-ufos-and-if-theyre-alien-in-origin Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 3:28 What is 'Oumuamua's wager? 11:29 The properties of 'Oumuamua and how they lend credence to the theory of it being artificial in origin 17:23 Theories of 'Oumuamua being natural in origin 21:42 Why was the smooth acceleration of 'Oumuamua significant? 23:35 What are comets and asteroids? 28:30 What we know about Oort clouds and how 'Oumuamua relates to what we expect of Oort clouds 33:40 Could there be exotic objects in Oort clouds that would account for 'Oumuamua 38:08 What is your credence that 'Oumuamua is alien in origin? 44:50 Bayesian reasoning and 'Oumuamua 46:34 How do UFO reports and sightings affect your perspective of 'Oumuamua? 54:35 Might alien artefacts be more common than we expect? 58:48 The Drake equation 1:01:50 Where are the most likely great filters? 1:11:22 Difficulties in scientific culture and how they affect fruitful inquiry 1:27:03 The cosmic endowment, traveling to galactic clusters, and galactic treaties 1:31:34 Why don't we find evidence of alien superstructures? 1:36:36 Looking for the bio and techno signatures of alien life 1:40:27 Do alien civilizations converge on beneficence? 1:43:05 Is there a necessary relationship between what is true and good? 1:47:02 Is morality evidence based knowledge? 1:48:18 Axiomatic based knowledge and testing moral systems 1:54:08 International governance and making contact with alien life 1:55:59 The need for an elite scientific body to advise on global catastrophic and existential risk 1:59:57 What are the most fundamental questions? This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Nicolas Berggruen, investor and philanthropist, joins us to explore the dynamics of power, wisdom, technology and ideas in the 21st century. Topics discussed in this episode include: -What wisdom consists of -The role of ideas in society and civilization -The increasing concentration of power and wealth -The technological displacement of human labor -Democracy, universal basic income, and universal basic capital -Living an examined life You can find the page for this podcast here: https://futureoflife.org/2021/05/31/nicolas-berggruen-on-the-dynamics-of-power-wisdom-technology-and-ideas-in-the-age-of-ai/ Check out Nicolas' thoughts archive here: www.nicolasberggruen.com Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 1:45 The race between the power of our technology and the wisdom with which we manage it 5:19 What is wisdom? 8:30 The power of ideas 11:06 Humanity’s investment in wisdom vs the power of our technology 15:39 Why does our wisdom lag behind our power? 20:51 Technology evolving into an agent 24:28 How ideas play a role in the value alignment of technology 30:14 Wisdom for building beneficial AI and mitigating the race to power 34:37 Does Mark Zuckerberg have control of Facebook? 36:39 Safeguarding the human mind and maintaining control of AI 42:26 The importance of the examined life in the 21st century 45:56 An example of the examined life 48:54 Important ideas for the 21st century 52:46 The concentration of power and wealth, and a proposal for universal basic capital 1:03:07 Negative and positive futures 1:06:30 Final thoughts from Nicolas This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Bart Selman, Professor of Computer Science at Cornell University, joins us to discuss a wide range of AI issues, from autonomous weapons and AI consciousness to international governance and the possibilities of superintelligence. Topics discussed in this episode include: -Negative and positive outcomes from AI in the short, medium, and long-terms -The perils and promises of AGI and superintelligence -AI alignment and AI existential risk -Lethal autonomous weapons -AI governance and racing to powerful AI systems -AI consciousness You can find the page for this podcast here: https://futureoflife.org/2021/05/20/bart-selman-on-the-promises-and-perils-of-artificial-intelligence/ Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 1:35 Futures that Bart is excited about 4:08 Positive futures in the short, medium, and long-terms 7:23 AGI timelines 8:11 Bart’s research on “planning” through the game of Sokoban 13:10 If we don’t go extinct, is the creation of AGI and superintelligence inevitable? 15:28 What’s exciting about futures with AGI and superintelligence? 17:10 How long does it take for superintelligence to arise after AGI? 21:08 Would a superintelligence have something intelligent to say about income inequality? 23:24 Are there true or false answers to moral questions? 25:30 Can AGI and superintelligence assist with moral and philosophical issues? 28:07 Do you think superintelligences converge on ethics? 29:32 Are you most excited about the short or long-term benefits of AI? 34:30 Is existential risk from AI a legitimate threat? 35:22 Is the AI alignment problem legitimate? 43:29 What are futures that you fear? 46:24 Do social media algorithms represent an instance of the alignment problem? 51:46 The importance of educating the public on AI 55:00 Income inequality, cyber security, and negative futures 1:00:06 Lethal autonomous weapons 1:01:50 Negative futures in the long-term 1:03:26 How have your views of AI alignment evolved? 1:06:53 Bart’s plans and intentions for the Association for the Advancement of Artificial Intelligence 1:13:45 Policy recommendations for existing AIs and the AI ecosystem 1:15:35 Solving the parts of the AI alignment that won’t be solved by industry incentives 1:18:17 Narratives of an international race to powerful AI systems 1:20:42 How does an international race to AI affect the chances of successful AI alignment? 1:23:20 Is AI a zero sum game? 1:28:51 Lethal autonomous weapons governance 1:31:38 Does the governance of autonomous weapons affect outcomes from AGI 1:33:00 AI consciousness 1:39:37 Alignment is important and the benefits of AI can be great This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Jaan Tallinn, investor, programmer, and co-founder of the Future of Life Institute, joins us to discuss his perspective on AI, synthetic biology, unknown unknows, and what's needed for mitigating existential risk in the 21st century. Topics discussed in this episode include: -Intelligence and coordination -Existential risk from AI, synthetic biology, and unknown unknowns -AI adoption as a delegation process -Jaan's investments and philanthropic efforts -International coordination and incentive structures -The short-term and long-term AI safety communities You can find the page for this podcast here: https://futureoflife.org/2021/04/20/jaan-tallinn-on-avoiding-civilizational-pitfalls-and-surviving-the-21st-century/ Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 1:29 How can humanity improve? 3:10 The importance of intelligence and coordination 8:30 The bottlenecks of input and output bandwidth as well as processing speed between AIs and humans 15:20 Making the creation of AI feel dangerous and how the nuclear power industry killed itself by downplaying risks 17:15 How Jaan evaluates and thinks about existential risk 18:30 Nuclear weapons as the first existential risk we faced 20:47 The likelihood of unknown unknown existential risks 25:04 Why Jaan doesn't see nuclear war as an existential risk 27:54 Climate change 29:00 Existential risk from synthetic biology 31:29 Learning from mistakes, lacking foresight, and the importance of generational knowledge 36:23 AI adoption as a delegation process 42:52 Attractors in the design space of AI 44:24 The regulation of AI 45:31 Jaan's investments and philanthropy in AI 55:18 International coordination issues from AI adoption as a delegation process 57:29 AI today and the negative impacts of recommender algorithms 1:02:43 Collective, institutional, and interpersonal coordination 1:05:23 The benefits and risks of longevity research 1:08:29 The long-term and short-term AI safety communities and their relationship with one another 1:12:35 Jaan's current philanthropic efforts 1:16:28 Software as a philanthropic target 1:19:03 How do we move towards beneficial futures with AI? 1:22:30 An idea Jaan finds meaningful 1:23:33 Final thoughts from Jaan 1:25:27 Where to find Jaan This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Joscha Bach, Cognitive Scientist and AI researcher, as well as Anthony Aguirre, UCSC Professor of Physics, join us to explore the world through the lens of computation and the difficulties we face on the way to beneficial futures. Topics discussed in this episode include: -Understanding the universe through digital physics -How human consciousness operates and is structured -The path to aligned AGI and bottlenecks to beneficial futures -Incentive structures and collective coordination You can find the page for this podcast here: https://futureoflife.org/2021/03/31/joscha-bach-and-anthony-aguirre-on-digital-physics-and-moving-towards-beneficial-futures/ You can find FLI's three new policy focused job postings here: futureoflife.org/job-postings/ Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 3:17 What is truth and knowledge? 11:39 What is subjectivity and objectivity? 14:32 What is the universe ultimately? 19:22 Is the universe a cellular automaton? Is the universe ultimately digital or analogue? 24:05 Hilbert's hotel from the point of view of computation 35:18 Seeing the world as a fractal 38:48 Describing human consciousness 51:10 Meaning, purpose, and harvesting negentropy 55:08 The path to aligned AGI 57:37 Bottlenecks to beneficial futures and existential security 1:06:53 A future with one, several, or many AGI systems? How do we maintain appropriate incentive structures? 1:19:39 Non-duality and collective coordination 1:22:53 What difficulties are there for an idealist worldview that involves computation? 1:27:20 Which features of mind and consciousness are necessarily coupled and which aren't? 1:36:40 Joscha's final thoughts on AGI This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Roman Yampolskiy, Professor of Computer Science at the University of Louisville, joins us to discuss whether we can control, comprehend, and explain AI systems, and how this constrains the project of AI safety. Topics discussed in this episode include: -Roman’s results on the unexplainability, incomprehensibility, and uncontrollability of AI -The relationship between AI safety, control, and alignment -Virtual worlds as a proposal for solving multi-multi alignment -AI security You can find the page for this podcast here: https://futureoflife.org/2021/03/19/roman-yampolskiy-on-the-uncontrollability-incomprehensibility-and-unexplainability-of-ai/ You can find FLI's three new policy focused job postings here: https://futureoflife.org/job-postings/ Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:35 Roman’s primary research interests 4:09 How theoretical proofs help AI safety research 6:23 How impossibility results constrain computer science systems 10:18 The inability to tell if arbitrary code is friendly or unfriendly 12:06 Impossibility results clarify what we can do 14:19 Roman’s results on unexplainability and incomprehensibility 22:34 Focusing on comprehensibility 26:17 Roman’s results on uncontrollability 28:33 Alignment as a subset of safety and control 30:48 The relationship between unexplainability, incomprehensibility, and uncontrollability with each other and with AI alignment 33:40 What does it mean to solve AI safety? 34:19 What do the impossibility results really mean? 37:07 Virtual worlds and AI alignment 49:55 AI security and malevolent agents 53:00 Air gapping, boxing, and other security methods 58:43 Some examples of historical failures of AI systems and what we can learn from them 1:01:20 Clarifying impossibility results 1:06 55 Examples of systems failing and what these demonstrate about AI 1:08:20 Are oracles a valid approach to AI safety? 1:10:30 Roman’s final thoughts This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Stuart Russell, Professor of Computer Science at UC Berkeley, and Zachary Kallenborn, WMD and drone swarms expert, join us to discuss the highest risk and most destabilizing aspects of lethal autonomous weapons. Topics discussed in this episode include: -The current state of the deployment and development of lethal autonomous weapons and swarm technologies -Drone swarms as a potential weapon of mass destruction -The risks of escalation, unpredictability, and proliferation with regards to autonomous weapons -The difficulty of attribution, verification, and accountability with autonomous weapons -Autonomous weapons governance as norm setting for global AI issues You can find the page for this podcast here: https://futureoflife.org/2021/02/25/stuart-russell-and-zachary-kallenborn-on-drone-swarms-and-the-riskiest-aspects-of-lethal-autonomous-weapons/ You can check out the new lethal autonomous weapons website here: https://autonomousweapons.org/ Have any feedback about the podcast? You can share your thoughts here: www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 2:23 Emilia Javorsky on lethal autonomous weapons 7:27 What is a lethal autonomous weapon? 11:33 Autonomous weapons that exist today 16:57 The concerns of collateral damage, accidental escalation, scalability, control, and error risk 26:57 The proliferation risk of autonomous weapons 32:30 To what extent are global superpowers pursuing these weapons? What is the state of industry's pursuit of the research and manufacturing of this technology 42:13 A possible proposal for a selective ban on small anti-personnel autonomous weapons 47:20 Lethal autonomous weapons as a potential weapon of mass destruction 53:49 The unpredictability of autonomous weapons, especially when swarms are interacting with other swarms 58:09 The risk of autonomous weapons escalating conflicts 01:10:50 The risk of drone swarms proliferating 01:20:16 The risk of assassination 01:23:25 The difficulty of attribution and accountability 01:26:05 The governance of autonomous weapons being relevant to the global governance of AI 01:30:11 The importance of verification for responsibility, accountability, and regulation 01:35:50 Concerns about the beginning of an arms race and the need for regulation 01:38:46 Wrapping up 01:39:23 Outro This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

John Prendergast, former adjunct professor of psychology at the California Institute of Integral Studies, joins Lucas Perry for a discussion about the experience and effects of ego-identification, how to shift to new levels of identity, the nature of non-dual awareness, and the potential relationship between waking up and collective human problems. This is not an FLI Podcast, but a special release where Lucas shares a direction he feels has an important relationship with AI alignment and existential risk issues. Topics discussed in this episode include: -The experience of egocentricity and ego-identification -Waking up into heart awareness -The movement towards and qualities of non-dual consciousness -The ways in which the condition of our minds collectively affect the world -How waking up may be relevant to the creation of AGI You can find the page for this podcast here: https://futureoflife.org/2021/02/09/john-prendergast-on-non-dual-awareness-and-wisdom-for-the-21st-century/ Have any feedback about the podcast? You can share your thoughts here: https://www.surveymonkey.com/r/DRBFZCT Timestamps: 0:00 Intro 7:10 The modern human condition 9:29 What egocentricity and ego-identification are 15:38 Moving beyond the experience of self 17:38 The origins and structure of self 20:25 A pointing out instruction for noticing ego-identification and waking up out of it 24:34 A pointing out instruction for abiding in heart-mind or heart awareness 28:53 The qualities of and moving into heart awareness and pure awareness 33:48 An explanation of non-dual awareness 40:50 Exploring the relationship between awareness, belief, and action 46:25 Growing up and improving the egoic structure 48:29 Waking up as recognizing true nature 51:04 Exploring awareness as primitive and primary 53:56 John's dream of Sri Nisargadatta Maharaj 57:57 The use and value of conceptual thought and the mind 1:00:57 The epistemics of heart-mind and the conceptual mind as we shift levels of identity 1:17:46 A pointing out instruction for inquiring into core beliefs 1:27:28 The universal heart, qualities of awakening, and the ethical implications of such shifts 1:31:38 Wisdom, waking up, and growing up for the transgenerational issues of the 21st century 1:38:44 Waking up and its applicability to the creation of AGI 1:43:25 Where to find, follow, and reach out to John 1:45:56 Outro This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Beatrice Fihn, executive director of the International Campaign to Abolish Nuclear Weapons (ICAN) and Nobel Peace Prize recipient, joins us to discuss the current risks of nuclear war, policies that can reduce the risks of nuclear conflict, and how to move towards a nuclear weapons free world. Topics discussed in this episode include: -The current nuclear weapons geopolitical situation -The risks and mechanics of accidental and intentional nuclear war -Policy proposals for reducing the risks of nuclear war -Deterrence theory -The Treaty on the Prohibition of Nuclear Weapons -Working towards the total elimination of nuclear weapons You can find the page for this podcast here: https://futureoflife.org/2021/01/21/beatrice-fihn-on-the-total-elimination-of-nuclear-weapons/ Timestamps: 0:00 Intro 4:28 Overview of the current nuclear weapons situation 6:47 The 9 nuclear weapons states, and accidental and intentional nuclear war 9:27 Accidental nuclear war and human systems 12:08 The risks of nuclear war in 2021 and nuclear stability 17:49 Toxic personalities and the human component of nuclear weapons 23:23 Policy proposals for reducing the risk of nuclear war 23:55 New START Treaty 25:42 What does it mean to maintain credible deterrence 26:45 ICAN and working on the Treaty on the Prohibition of Nuclear Weapons 28:00 Deterrence theoretic arguments for nuclear weapons 32:36 The reduction of nuclear weapons, no first use, removing ground based missile systems, removing hair-trigger alert, removing presidential authority to use nuclear weapons 39:13 Arguments for and against nuclear risk reduction policy proposals 46:02 Moving all of the United State's nuclear weapons to bombers and nuclear submarines 48:27 Working towards and the theory of the total elimination of nuclear weapons 1:11:40 The value of the Treaty on the Prohibition of Nuclear Weapons 1:14:26 Elevating activism around nuclear weapons and messaging more skillfully 1:15:40 What the public needs to understand about nuclear weapons 1:16:35 World leaders' views of the treaty 1:17:15 How to get involved This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.

Max Tegmark and members of the FLI core team come together to discuss favorite projects from 2020, what we've learned from the past year, and what we think is needed for existential risk reduction in 2021. Topics discussed in this episode include: -FLI's perspectives on 2020 and hopes for 2021 -What our favorite projects from 2020 were -The biggest lessons we've learned from 2020 -What we see as crucial and needed in 2021 to ensure and make -improvements towards existential safety You can find the page for this podcast here: https://futureoflife.org/2021/01/08/max-tegmark-and-the-fli-team-on-2020-and-existential-risk-reduction-in-the-new-year/ Timestamps: 0:00 Intro 00:52 First question: What was your favorite project from 2020? 1:03 Max Tegmark on the Future of Life Award 4:15 Anthony Aguirre on AI Loyalty 9:18 David Nicholson on the Future of Life Award 12:23 Emilia Javorksy on being a co-champion for the UN Secretary-General's effort on digital cooperation 14:03 Jared Brown on developing comments on the European Union's White Paper on AI through community collaboration 16:40 Tucker Davey on editing the biography of Victor Zhdanov 19:49 Lucas Perry on the podcast and Pindex video 23:17 Second question: What lessons do you take away from 2020? 23:26 Max Tegmark on human fragility and vulnerability 25:14 Max Tegmark on learning from history 26:47 Max Tegmark on the growing threats of AI 29:45 Anthony Aguirre on the inability of present-day institutions to deal with large unexpected problems 33:00 David Nicholson on the need for self-reflection on the use and development of technology 38:05 Emilia Javorsky on the global community coming to awareness about tail risks 39:48 Jared Brown on our vulnerability to low probability, high impact events and the importance of adaptability and policy engagement 41:43 Tucker Davey on taking existential risks more seriously and ethics-washing 43:57 Lucas Perry on the fragility of human systems 45:40 Third question: What is needed in 2021 to make progress on existential risk mitigation 45:50 Max Tegmark on holding Big Tech accountable, repairing geopolitics, and fighting the myth of the technological zero-sum game 49:58 Anthony Aguirre on the importance of spreading understanding of expected value reasoning and fixing the information crisis 53:41 David Nicholson on the need to reflect on our values and relationship with technology 54:35 Emilia Javorksy on the importance of returning to multilateralism and global dialogue 56:00 Jared Brown on the need for robust government engagement 57:30 Lucas Perry on the need for creating institutions for existential risk mitigation and global cooperation 1:00:10 Outro This podcast is possible because of the support of listeners like you. If you found this conversation to be meaningful or valuable, consider supporting it directly by donating at futureoflife.org/donate. Contributions like yours make these conversations possible.