Nature of Intelligence, Ep. 5: How do we assess intelligence?
When it comes to assessing intelligence, people have all kinds of tests — the SAT, IQ tests, and so on. There’s controversy over how fairly these tests really measure human intelligence, but at the very least, we know that they correlate with some general reasoning skills when people take them. That assumption breaks down when we try to assess intelligence in non-humans. What does it mean when a large language model passes an intelligence test meant for humans? Does it actually have the same reasoning skills that a human does, or is it doing something else? In today’s episode, with guests Erica Cartmill and Ellie Pavlick, we investigate the best ways to assess intelligence in non-humans, whether animals or machines.
Show Notes
Guests:
- Erica Cartmill, Professor, Anthropology and Cognitive Science, Indiana University Bloomington
- Ellie Pavlick, Assistant Professor, Computer Science and Linguistics, Brown University
Hosts: Abha Eli Phoboo & Melanie Mitchell
Producer: Katherine Moncure
Podcast theme music by: Mitch Mignano
Follow us on:
Twitter • YouTube • Facebook • Instagram • LinkedIn • Bluesky
More info:
- Tutorial: Fundamentals of Machine Learning
- Lecture: Artificial Intelligence
- SFI programs: Education
- Diverse Intelligences Summer Institute
Books:
- Artificial Intelligence: A Guide for Thinking Humans by Melanie Mitchell
Talks:
- How do we know what an animal understands by Erica Cartmill
- The Future of Artificial Intelligence by Melanie Mitchell
Papers & Articles:
- “Just kidding: the evolutionary roots of playful teasing,” in Biology Letters (September 23, 2020), doi.org/10.1098/rsbl.2020.0370
- “Overcoming bias in the comparison of human language and animal communication,” in PNAS (November 13, 2023), doi.org/10.1073/pnas.22187991
- “Using the senses in animal communication,” by Erica Cartmill, in A New Companion to Linguistic Anthropology, Chapter 20, Wiley Online Library (March 21, 2023)
- “Symbols and grounding in large language models,” in Philosophical Transactions of the Royal Society A (June 5, 2023), doi.org/10.1098/rsta.2022.0041
- “Emergence of abstract state representations in embodied sequence modeling,” in arXiv (November 7, 2023), doi.org/10.48550/arXiv.2311.02171
- “How do we know how smart AI systems are,” in Science (July 13, 2023), doi: 10.1126/science.adj59