BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices

This research paper presents a framework for assessing the quality of AI benchmarks, which are tools used to measure the performance of artificial intelligence models. The authors identify several best practices for benchmark development across five stages of a benchmark's lifecycle: design, implementation, documentation, maintenance, and retirement. The framework and checklist are designed to help benchmark developers produce higher-quality benchmarks, leading to more reliable and informative evaluations of AI models.

https://arxiv.org/pdf/2411.12990