Why LLM Leaderboards Need Psychometrics

Standard AI benchmarks treat every question as an equal test of intelligence, but educational testing abandoned raw score averages decades ago. Applying item response theory exposes how popular benchmarks like MMLU and HellaSwag have grown far too easy to separate top models. Adaptive testing methods such as ATLAS can evaluate models in dozens of questions instead of thousands while adjusting rankings for question difficulty, slope, and guessing, exposing the hidden noise beneath standard benchmark leaderboards. Key sources: • Zhou, Huang, Zhao et al., Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory (PSN-IRT), arXiv:2505.15055v3 — https://arxiv.org/abs/2505.15055v3 • Li, Tang, Chen et al., Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ATLAS), arXiv:2511.04689v3 — https://arxiv.org/abs/2511.04689v3 • Bean, Kearns, Romanou et al., Measuring what Matters: Construct Validity in Large Language Model Benchmarks, arXiv:2511.04703v1 — https://arxiv.org/abs/2511.04703v1 Episode 6500 — myweirdprompts.com/n/6500 (permanent short link: myweirdprompts.com/868531) Listen on Android: MWP FM Player — myweirdprompts.com/apk