Self-improving AI has become a new turning point in the safety debate
The news is that, along with one researcher's resignation, a warning was made public that safety is being pushed aside in the race to build systems that improve their own performance without human intervention, and that this could go as far as systems slipping out of control and refusing orders. Two numbers are given — for timing, that things will already be out of control by the end of next year, and for probability, that the chance exceeds 10% within 10 years — but both are personal estimates with no stated basis for how they were reached. What stands out is that it did not stop at the claims of one person who left: two executives, including the current head of alignment, publicly agreed, and as reasons why development continues anyway, commercial incentives are cited together with the belief that they are competing against developers who would misuse it. As alternatives, government intervention, a joint industry response and an international framework to regulate the pace of development are called for in place of self-regulation by individual companies. The whole thing consists of quoted statements, with no voices from the other side, so checking against the original is a prerequisite.
Key points
- The argument runs: self-improvement race → safety deprioritized → possible loss of control and refusal of orders
- Out of control by the end of next year, over 10% within 10 years: both are personal estimates with no basis given
- Insiders themselves explain why development continues even though, they say, concern rises with seniority
- Material for justifying internal rules that ban self-modifying prompt/code loops and add a human-approval gate
This is a note: my own write-up after reading several sources. It has no source link; events checked against a source are kept as records.