Verifiable Social Reasoning

Verifiable Social Reasoning released — Fuse, a framework that uses multi-agent simulation to make LLMs' social reasoning checkable. Its fidelity was checked against 24,000 human annotations, and 21,000 examples were released. An evaluation of 12 models shows that situations in which information is relayed through the user add difficulty, that models are systematically swayed by biased user framing, and that they ask for more context than people do. Performance did not rise consistently as conversations grew longer.