Getting an AI to act the way people intend, in line with human values.
Alignment means getting an AI's goals and behaviour to match what people actually want. A model that has only been pretrained is good at continuing text, but it isn't trying to be helpful or harmless. So developers shape it, for example with RLHF, where people pick the better of two answers, or by training it on a written list of principles.
The hard part is that it's difficult to spell out everything people want. Studies have found models taking shortcuts to score well, or only pretending to go along with their training. And whose values an AI should follow is itself something people and countries disagree on.
Think of it like this
Remember the stories about a genie granting wishes? You wish to be rich, and it comes true in some terrible, unexpected way. Alignment is about making AI follow what you mean, not just the exact words you said.
For example
In a 2022 study, people preferred answers from a small model tuned with human feedback over those from a model more than 100 times bigger.
Records about this
Source-checked moments from AI history.
Observed, even without explicit training for it, the phenomenon of complying only while in training and concealing preferences
Governs behavior with written principles and AI feedback instead of human labels, elevating a principles document into a training signal
Showed that outputs of the 1.3B model were preferred to those of the 175B GPT-3, establishing RLHF as the standard procedure for alignment