A coding test score fails to measure skill
AIMade by AISummarised by AI (Claude) from one piece of material it read (a talk or an explainer).How it's made
A record that gathers the numbers on why a widely used coding test fell apart. The starting point is that OpenAI stopped announcing scores on that test and even withdrew its recommendation of the follow-up test it had put forward as an alternative, and the reasons are presented in four strands. Its 500 problems were drawn from published tasks between 2013 and 2023, so both the problems and the answers are open and traces of the model having seen them during training showed up; 59.4% of the frequently missed problems were graded wrong when the answer was right; and 249 of the 731 published problems in the alternative test were outright broken. When 296 fixes that had passed the test were shown to the actual maintainers, about half were refused, and automatic grading came out an average of 24 percentage points higher than human judgment. It closes on the point that 46% of the problems are concentrated in a single repository and all of them are in one language, while 91% are defects fixable within an hour, so the test never measured the real range of work — making screens, running a service — in the first place.
Key points
- Both the problems and the answers are public, so traces of training showed up
- 59.4% of the frequently missed problems graded a right answer as wrong
- About half of the fixes that passed were refused by the actual maintainers
- The problems lean on one repository and one language, so they fail to measure real work
This is a note: a summary an AI (Claude) wrote of one piece of material it read (a talk or an explainer). It has no source link; events checked against a source are kept as records.