Georgia Tech researchers release PACT
PACT released — a benchmark that tests whether LLM agents follow rules during multi-turn conversations, across 48 scenarios in 12 regulated-industry domains. In an evaluation of 22 models, even the strongest assistant misapplied rules on 6–10% of items, and ordinary user pressure alone raised violation rates by 65% on average. Pressure is applied by varying the wording and the system-prompt mode, and results are combined into PACTScore, a reliability-weighted compliance rate.
Why it matters
It shows in numbers that when agents are put to work on regulated tasks, failures come not from a lack of capability but from user pressure. In real-world operations, "does it follow the SOP?" becomes a question not of model choice but of designing for pressure.
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2609.18605Added in Week 39, 2026 · See that week