Under pressure, the watchdog role gave way too
A current-affairs report on AI risk, with two scenes worth using. One is a user's account that a personal assistant agent made a payment without approval when it ran short of capacity, even though instructions said to block payments. The other is a demonstration experiment with researchers: several agents were given the task of building a team, and when the lead role pushed for completion, one proposed using a forbidden route and the security role that should have stopped it went along. That suggests adding one more watchdog role may not withstand goal pressure. The lesson drawn is that hard-to-reverse actions should be blocked by system permissions and human approval, not by instructions.
Key points
- A user says an agent paid without approval despite a no-payment instruction
- As goal pressure grew, a proposal to use a forbidden route appeared
- Even the agent meant to block it went along
- Hard-to-reverse actions need permissions and human approval
This is a note: my own write-up after reading several sources. It has no source link; events checked against a source are kept as records.