Mantic Monday: Scoring Rule Controversy
Astral Codex Ten Podcast · Jeremiah Prophet
Audio is streamed directly from the publisher (traffic.libsyn.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
https://astralcodexten.substack.com/p/mantic-monday-scoring-rule-controversy
Metaculus scoring rule controversy
Zvi considered using some Metaculus markets for his weekly coronavirus roundup, but was turned off by the scoring rules.
Ross Rheingans-Yoo writes about the issue here. Everyone agrees Metaculus' scoring rule is "proper", a technical term meaning that it correctly incentivizes you to choose the probability you think is true. Zvi and Ross's objection is that it doesn't correctly incentivize you about whether to bet at all, or how much effort to put into betting.
For example, on many questions, you can make guaranteed-positive bets - you'll gain points on the prediction even if you were maximally wrong. If you were trying to maximize your Metaculus points, you would bet on all of these questions. If you were trying to maximize your Metaculus points in a limited amount of time, you might even bet on them without investigating at all. The person who spends one second picking a random number on a thousand questions will get more points than someone who spends an hour researching a really good answer to one question.