Scale AI

Quantitative evidence of a failure to transfer to real-world knowledge work

A follow-up design that responds to contamination and saturation in the SWE-bench family. An attempt to prevent leakage structurally by separating tiers

The next frontier after MMLU saturation; models at the time scored under 10%

Measured benchmark contamination with a controlled experiment. However, reported that frontier models show few signs of overfitting