Episode 99: Is MTurk Too Good To Be True?
Alexa and Yoel weigh in on recent debates about whether psychological researchers can get good data online. They consider criticisms and defenses of online participant-recruitment platforms like Amazon Mechanical Turk and throw a bit of their own experience into the mix.
Audio is streamed directly from the publisher (aphid.fireside.fm) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
In a recent article, psychologists Webb and Tangney document their experience collecting psychology data online using Amazon's crowdsourcing platform MTurk. Alarmingly, the authors conclude that ultimately only 2.6% of their sample was valid data from human beings. Yoel and Alexa weigh in on these findings, discussing what researchers can reasonably expect from online studies and platforms, and how their personal experiences have informed their own practices. They also consider a response written by Cuskley and Sulik, who argue that researchers, not recruitment platforms, are responsible for ensuring the quality of data collected online. Questions that arise include: What studies do people want to do? Does anyone read the fine print? And what are the ethics of mouse-hunting?
Links:
- Two Psychologists Four Beers on Untappd
- Too Good to Be True: Bots and Bad Data From Mechanical Turk - Margaret A. Webb, June P. Tangney, 2022
- PsyArXiv Preprints | The burden for high-quality online data collection lies with researchers, not recruitment platforms
- Living in Harmony with House Mice and Rats | PETA