A shared test that gives AI models the same questions so their scores can be compared.
A benchmark is a test that gives many AI models exactly the same questions and compares their scores. Different benchmarks measure different skills: MMLU has multiple-choice questions across 57 subjects, while SWE-bench asks models to fix real problems reported in software projects. When a new model comes out, its makers usually publish scores like these.
Scores don't tell the whole story. If test questions leak into the training data, a model may have memorised the answers rather than worked them out. And once models score close to full marks, the test can no longer tell them apart, so researchers build harder ones.
Think of it like this
Think of a big exam that everyone takes. Because everyone sits the same paper, the scores can be compared. But if someone saw the questions beforehand, a high score doesn't prove much.
For example
In 2019, after AI scores passed non-expert humans on the GLUE test, researchers replaced it with the harder SuperGLUE.
Records about this
Source-checked moments from AI history.
Measured benchmark contamination with a controlled experiment. However, reported that frontier models show few signs of overfitting
Became the de facto standard for measuring the breadth of knowledge of pretrained models, and an essential metric in model announcements for the next 5 years
The first instance of the cycle from benchmark saturation to replacement