Sutton's diagnosis: an LLM whose weights never change is not a mind

The gist of what Rich Sutton and his student Khurram Javed say is that the path current AI is on is fundamentally wrong. The core of the Bitter Lesson is to stop clinging to human knowledge and focus on methods that scale with computation, such as search and learning; LLMs are both a positive example, having scaled computation by absorbing the entire internet, and a negative example, running into limits because they depend on finite human knowledge. Synthetic data is a mistake trapped within the limits of human knowledge, since in the end people decide what good data is; and under the big world hypothesis — that the world is overwhelmingly more complex than any agent — their position is that only systems that keep learning from experience are the answer. The real problem is that weights do not change at all after deployment; they see catastrophic forgetting as curable with per-weight step-size meta-learning and continual backpropagation, which keeps planting new units, and Oak Lab is aiming for a model with 1 trillion parameters running on 20 watts within 5–10 years. The assessment that language ability is about a quarter of intelligence and the diagnosis that the big labs are stuck in a local optimum come from a secondary summary, so they need to be checked against the original interview.

This is a note: my own write-up after reading several sources. It has no source link; events checked against a source are kept as records.