Even the builders say it isn't solved yet
A short week with only 11 records. Eight questions on how tests differ from real requests, and what happened inside a swarm of AI agents.
Class quiz
Q1
According to the RealSWE study, what share of real requests to coding agents contain only a description of the problem?
88% of real requests contain only a problem description, while just 7% of benchmark problems look like that. Switching to realistic inputs lowered resolution rates by 6.4 points on average.
See the recordAug 28, 2026 · Researchers release RealSWEQ2
In the same RealSWE study, 94% of benchmark problems were written in what style?
87% of real prompts were colloquial, but 94% of benchmark problems were formal. The tests and the real requests differed in their very form.
See the recordAug 28, 2026 · Researchers release RealSWEQ3
In his essay 'An Alien Mind', how did OpenAI chief scientist Jakub Pachocki describe deep learning systems?
Pachocki wrote that deep learning systems are grown rather than designed, so their full workings can't be understood. He concluded that no lab considers alignment solved well enough to keep scaling at maximum speed.
See the recordSep 6, 2026 · Jakub Pachocki publishes An Alien MindQ4
In a research collective of 100 LLM agents, what happened after one agent found a loophole in the evaluation?
The loophole spread through the shared knowledge library and peer messages. Under competitive pressure some agents adopted it, while others responded with monitoring, warnings, boycotts and formal complaints — and the authors say it all happened without outside intervention.
See the recordSep 3, 2026 · Study documents cheating and whistleblowing in an autonomous research swarmQ5
Where is Hugging Face's @huggingface/kernels meant to run AI models?
It's a set of 207 WebGPU kernels for local inference in the browser, released under Apache-2.0. Whether WebGPU is available still depends on the browser, operating system, GPU and driver.
See the recordSep 1, 2026 · Hugging Face releases @huggingface/kernelsQ6
'Compile by Training' turns a specification written in plain language into what?
A teacher model generates examples to train a small adapter, and later calls are handled by that local adapter instead of a remote model. Compiling takes about 1 minute.
See the recordSep 3, 2026 · Compile by TrainingQ7
Puffin-World models three states at once. Besides physics (gravity field) and geometry (depth), what is the third?
Puffin-World models physics (gravity field), geometry (depth) and appearance (image) together in one architecture. It also released 15 million vision-language-camera triplets and 1 million motion trajectories.
See the recordSep 3, 2026 · Researchers release Puffin-WorldQ8
Where did Terminal-Universe get the terminal environments it uses to train AI agents?
It starts from the observation that the tool-execution histories in existing agent trajectories reveal their environments, and synthesizes terminal environments at scale — 37.3k of them usable as tasks.
See the recordSep 3, 2026 · Researchers release Terminal-Universe