A report that used numbers obtained by asking models as evidence

This sums up a short report that bundles together a safety warning and a performance breakthrough from the same week. Its construction is unusual: it directly asks two commercial models whether they are confident that their own companies are developing them responsibly and uses the answers as evidence, and goes further, asking them for the probability of human extinction within 10 years, getting figures of 5–10% and 1–5%, and listing them on the same level as expert estimates. These numbers are neither measurements nor survey results but sentences generated by models, so the report is better read as a case showing where attempts to use answers obtained from models as evidence go wrong. As a contrasting event, it adds an announcement that an unreleased next-generation model found, in 8 hours, a solution to a mathematical problem that had gone unsolved for more than 90 years, which is linked to the concern that models have reached the point of developing their own successors. The point raised in one misuse report, that telling therapeutic purposes from weapons-development purposes is especially difficult, can be read as a structural limit: models cannot tell the intent behind a query.

This is a note: my own write-up after reading several sources. It has no source link; events checked against a source are kept as records.