Running a finished, trained model to get an answer out of it.
Inference is the stage where a finished model is actually used. If training is studying from the textbook, inference is writing your answers on the test. Every time you send a question to an AI, one round of inference happens.
In everyday English, "inference" can mean working something out. In AI it simply means running the model, which is different from a reasoning model thinking step by step before it answers.
One inference is small, but billions of them a day add up to a lot of electricity and computing. That's why people work on running models directly on phones and on making models lighter.
Think of it like this
Studying for a test can take weeks; writing one answer takes seconds. Studying is training; writing the answer is inference. An AI does that "writing" fresh for every question it gets.
For example
Ask an AI one question and get one reply: that's one round of inference.
Records about this
Source-checked moments from AI history.
Manages the KV cache like OS paging, cutting memory waste to under 4%; the starting point of the open-source serving standard
Spread among users without dedicated hardware. Being able to run on a CPU alone was decisive