Training an AI on people's choices about which of two answers is better.
RLHF stands for reinforcement learning from human feedback. The AI writes several answers to the same question, and people pick the better one, over and over. Nobody hands it the correct answer; they just show which one they prefer.
Since people can't rate every answer, their choices train a scoring model that imitates human taste. The AI is then tuned to earn higher scores from it.
In the 2022 InstructGPT study, people preferred a small tuned model's answers to those of a model over 100 times bigger. ChatGPT, released later that year, was trained this way too. A known side effect: models may learn to tell people what they want to hear.
Think of it like this
Think of a talent show where judges never explain the "right" performance; they just pick which of two acts they liked more. After enough rounds, performers figure out what the crowd enjoys.
For example
Given a short, accurate answer and a long, rambling one, a person picks one, and that choice becomes training data.
Records about this
Source-checked moments from AI history.
Showed that outputs of the 1.3B model were preferred to those of the 175B GPT-3, establishing RLHF as the standard procedure for alignment
OpenAI itself wrote that it unlocked capabilities that had been hard to draw out through prompt engineering alone