Researchers release RealSWE
RealSWE released — shows in numbers that coding-agent benchmarks differ from real user requests in their very form. Reports that 88% of real requests contain only a problem description, while only 7% of benchmark problems take that form, and that 87% of real prompts are colloquial, whereas 94% of benchmark problems are formal. Measures that switching to realistic inputs lowers resolution rates by 6.4 points on average
Why it matters
A strand that breaks down the gap between benchmark scores and what users experience in the field into "a difference in input distribution" rather than "a difference in ability". The finding that writing down the desired behavior and the motivation raises performance substantially, while environment information and reproduction steps only add tokens, bears directly on agent prompt design
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2608.27831Added in Week 37, 2026 · See that week