Part 1 Review
The real terms, defined and then used · down to the mechanics and backstory · DAY 01–10
Here we use the real terms — words like autoregressive, softmax, loss, backpropagation, context length and distillation.
Each one gets a one-sentence explanation the first time it shows up, and after that I'll use it as is, no shortcuts. The facts are the same as in Beginner and Intermediate — only how deep we dig is different.
DAY 01 · Guessing the next word
Pick a piece, add it, pick again
AI doesn't make up a whole sentence at once. It picks just the next piece, adds it, then picks again. Repeat that pick hundreds of times, and you get the long answer we read. So AI isn't fetching an answer stored somewhere — it builds a new one every time.
Because that's exactly the job it got in training — cover part of a sentence and guess the next piece. A machine that learned to guess well works the same way when it answers. That's also why answers seem to trickle out bit by bit. That really is how they're being made.
This is called autoregressive generation — each piece it just made goes back into the input to pick the next. The weak spot: it never goes back. Pick one odd piece early on, and it keeps building on top of it. Most language models today stand on this setup. What's changed is how they pick — not the fact that they pick.
DAY 02 · Tokens
That piece has a name: token
The piece AI picks has a name. It's called a token. A token isn't a letter or a word — common chunks stay whole, and unfamiliar ones get chopped up small. To you, 'What's for lunch today?' is four words. To an AI, it's about six tokens.
The cutting rules are called a tokenizer. Each model has its own, so token counts for one sentence vary. More tokens means more math to do — so it gets slower and pricier. Languages like Korean often need more tokens than English to say the same thing.
BPE (Byte Pair Encoding) is a common method: it builds a vocabulary by merging frequent pairs, over and over. So even a never-seen word can be built from small pieces — words not in the vocabulary don't stump it. The 'context length' coming up later is counted in tokens, not letters — tokens are the unit of all the math.
DAY 03 · Probability and temperature
The probability table and the temperature knob
To pick the next piece, AI scores every candidate, then holds a raffle where higher scores get more tickets. Temperature is the knob that sets how adventurous that raffle gets. Turn it down and #1 wins every time; turn it up and lower candidates pop up more often.
Same question, different answer each time? That's not a mood — it's the raffle. For jobs with one right answer (math, summaries, code), turn it down; when you want lots of ideas, turn it up. Turning it up doesn't make it smarter — just more varied, and more likely to say something off the wall.
Softmax turns scores into probabilities. Temperature divides scores first; higher means smaller gaps, flatter table. Set temperature close to 0, and it basically just picks #1 every time. Other knobs come along: top-k keeps just the top few; top-p keeps candidates until their odds hit a set total.
DAY 04 · Training
Training is tiny fixes, repeated
AI training isn't a sudden 'aha' — it's 'tiny fixes,' repeated. Hide part of a sentence, have it guess the next piece, then nudge the numbers inside by how much it missed. Repeat that an enormous number of times, and grammar and common sense soak into those numbers.
It's close to giving every miss an 'off by this much' score, then shifting a little in that direction. No big fixes all at once. Small ones — but a huge number of times. That's why training takes huge amounts of time and electricity — and once it's done, the numbers are locked.
Loss measures how wrong it was. Backpropagation works backward to tweak numbers and cut it — set out in 1986. Once training is over, chatting with us doesn't teach the model anything. Those numbers stay put. So 'Remember what I told you yesterday?' doesn't work. What looks like memory is another mechanism — DAY 07.
DAY 05 · Parameters
Parameters: how many numbers get tuned
How many numbers training tunes — that's the parameter count. The more there are, the bigger the container for patterns. Today's big models run from tens of billions to around 1 trillion — not a number you could ever count.
Counting one per second, 100 billion would take you more than 3,000 years. The human brain has about 100 trillion synapses, so models still fall short of that. But bigger doesn't always mean smarter — what it trained on, and how much, has to come with it.
In 2020, the GPT-3 paper made 175 billion the yardstick for this race. In 2022, the Chinchilla study showed balancing size with the amount of training data beats just scaling up. So now people talk more about 'how was it trained?' than 'how many?'
DAY 06 · Hallucination
Confidently making things up — hallucination
When AI calmly makes up facts that don't exist, it's called hallucination. It isn't trying to lie. It comes naturally with a setup that picks a plausible next piece. AI has no separate built-in way to flag 'I don't know this.'
It slips most on things rare in its training data, very recent events, and specifics: numbers, dates, names. It sounds the same whether it knows or not. Looking confident doesn't make it true. The habit of asking for sources and opening them yourself is the cheapest, surest defense.
A 2023 survey sorts hallucinations in two: clashing with the input vs. clashing with real-world facts. No way to fully remove it yet — people shrink it by adding search, requiring sources, or scoring confidence. So asking in a way you can check — that's the skill on the user's side.
DAY 07 · Memory limits
Not remembering — rereading
AI doesn't save your chat. Every time, it rereads the whole conversation so far, then answers. There's a limit to how much it can read at once. That limit is called the context length. As the chat gets longer, the early part gets pushed out of the window.
Sometimes 'I already told you!' doesn't work. What you said may already be out the window. Even with room to spare, there's a catch — studies found the middle tends to count less than the start or end. For really important info, the surest move is to point it out again later in the chat.
This window is measured in tokens (from DAY 02), not letters. The 2023 'Lost in the Middle' study showed that info in the middle of a long input gets used less. However long the window gets, reading isn't free — the longer it is, the slower and pricier.
DAY 08 · Prompts
New question, new probability table
A prompt is the whole input you hand to AI. Change the words, and the probability table for the next piece changes too — so the answer changes a lot. You're not changing the AI — you're changing the question.
Give it a role, spell out the conditions, name the format you want — these three work best. 'A 100-word notice for middle schoolers, with one example' gets way better answers than 'Write something good.' It's not a trick. You're just saying what you want more clearly.
A 2021 survey described prompts as an interface for getting the behavior you want without retraining. Sometimes just changing how you ask — like having it solve step by step — changes how accurate it is. But a prompt can't erase the hallucination from DAY 06 — it can only shrink it.
DAY 09 · Big models, small models
Size means parameter count
A 'big' model just means it has lots of parameters — the ones from DAY 05. Big models know a lot and can handle hard problems, but they're slow, pricey and only run on servers. Small models fit on your own device, so they're fast and work even without internet.
Quick, frequent, light jobs go small; jobs that need long thinking and lots of reading go big. You can also teach a small model, using a big model's answers as its textbook. Then the small model punches above its weight — but it can't fully replace the big one.
This is called distillation, and Geoffrey Hinton's team formalized it in 2015. A lot of the AI running on phones today came this way. It's not better vs. worse, it's placement — where it'll be used decides the right size.
DAY 10 · Open and closed models
Where the core lives
A closed model's core stays on the company's servers. We can only talk to it through the front desk. With an open model, you can download that core — the weights — in full and run it on your own computer. You ask questions the same way — the difference is where the model lives.
'Open' isn't one thing. Split it into four: weights, code to run it, training data, the right to change it. Most open up the weights and code, but not the training data. So don't stop at the word 'open' — ask which pieces are actually open.
Llama 2 (2023) even allowed commercial use, but its custom license sparked an 'is this open source?' debate. The same year, Mistral 7B went with the widely used Apache 2.0 license as is, dropping nearly all limits. 'Open' doesn't mean free — the hardware to run it, and the updates, are on whoever downloads it.
Ten things that follow from one design
A language model is an autoregressive generator. It picks one piece (a token) from a probability table, adds it on, and feeds the result back in as input. Training is what built the probability table, and how many numbers training adjusted is the parameter count.
Hallucination comes from this design being a probability generator, not a fact-checker. Memory limits come from rereading everything each time instead of storing it — and from that window being finite. Both are less defects than shadows of the design.
We get three knobs to hold — changing the input (prompts), choosing the size (big vs. small models), and choosing where to keep and run it (open vs. closed models). In Part 2, we look at how this design got here.
Learn more
- Attention Is All You Need (2017) — the Transformerarxiv.org/abs/1706.03762
- Neural Machine Translation of Rare Words with Subword Units (2015) — the BPE tokenizerarxiv.org/abs/1508.07909
- Learning representations by back-propagating errors (1986) — backpropagationnature.com/articles/323533a0
- Language Models are Few-Shot Learners (2020) — GPT-3, 175 billion parametersarxiv.org/abs/2005.14165
- Training Compute-Optimal Large Language Models (2022) — Chinchillaarxiv.org/abs/2203.15556
- A Survey on Hallucination in Large Language Models (2023)arxiv.org/abs/2311.05232
- Lost in the Middle (2023) — the middle of a long input gets used lessarxiv.org/abs/2307.03172
- Pre-train, Prompt, and Predict (2021) — a survey of promptingarxiv.org/abs/2107.13586
- Distilling the Knowledge in a Neural Network (2015) — distillationarxiv.org/abs/1503.02531
- Llama 2 release (2023) — open weights and the license debateen.wikipedia.org/wiki/Llama_(language_model)
- Mistral 7B (2023) — Apache 2.0mistral.ai/news/announcing-mistral-7b