Evaluation & BenchmarksDongxu Yang et al.
FDE-Gym released
FDE-Gym released — a benchmark modeled on the work of an engineer posted to a customer, with 95 delivery tasks drawn from 20 situations across 18 industries that have to be solved while asking after vague requirements and constraints nobody stated
Why it matters
The authors reported that in the setting where the constraints have to be found by the model, even the best configuration averaged only 0.56 points and solved just 14 of the 95 tasks. Even in the 23 tasks where every necessary fact was obtained from the stakeholder, only 10 scored 0.8 or above, and they concluded that finding the unstated constraints is the bottleneck
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2610.22350Added in Week 43, 2026 · See that week