Instead of memorizing each technology one by one, this page follows what the problem was and how it got solved. Each era starts from a problem the era before it could not solve and left behind. Once you see those links, you can also guess where a new technology will fit.
1,295records checked
1943–2026years covered
22axes
87%from primary sources
Four units
Read in order and each later idea is explained by the earlier ones. Unit ① sets up the frame, ② sweeps through the eras, ③ narrows to the last three years, and ④ sorts out the vocabulary.
The hierarchy of terms, the one task a language model solves, the cost structure, the four pieces, and how this material is organized.
01
Artificial intelligence · machine learning · deep learning · generative AI
The first three nest inside one another. But generative AI is a term cut by a different criterion, so it doesn't fit into the hierarchy.
AI ⊃ machine learning ⊃ deep learning is a true nesting. An expert system whose rules are written by hand is AI but not machine learning, and decision trees and SVMs are machine learning but not deep learning.
Generative AI is not a level in this hierarchy. It uses a different criterion — the term is defined not by method but by the nature of the output. If a model produces new data instead of classifying or scoring, it is generative.
That's why they overlap. Some deep learning isn't generative (a classifier that flags defective parts), and the same language model can be used both to generate and to classify. The question "which is broader, deep learning or generative AI?" doesn't even make sense.
Deep learning's original claim was not about performance but about representation learning — instead of people designing the features, the model builds useful representations on its own. That was the core of the 1986 paper (see Era 01).
Common mistakes"AI = deep learning" is wrong. And "generative AI" is not a subset of deep learning but a name based on a different criterion. Keep just these two apart, and half the articles you read will look different.
Experiment
What sits inside what
Press the buttons. The first three are nested inside each other; only generative AI uses a different criterion, so it overlaps them with a dashed line. That's why the question “which is broader, generative AI or deep learning?” doesn't make sense.
02
The one task a language model actually solves
Translation, summarizing and coding weren't trained separately. They all come from one thing: guessing the next token.
There is only one training task: look at the tokens so far and predict the probability distribution of the next token. No person needs to attach answer labels — the text itself contains the answers. This property is what made web-scale training possible.
What makes tasks emerge from this is the shape of the data. The internet is already full of "question → answer," "original → translation" and "code → explanation" pairs. To predict the next token well, a model has no choice but to model those relationships.
When generating, the model samples from that distribution. Temperature is the knob for how sharp (low: always the most likely option) or flat (high: more varied, but with a risk of nonsense) to make the distribution. This is why the same question gets a slightly different answer each time.
But this alone is not enough. A model trained only on next-token prediction may, instead of answering a question, produce more questions like it. Making it follow instructions takes post-training — that's the story of Era 03.
May 28, 2020OpenAI (Brown et al.) — GPT-3 — performs new tasks from examples in the prompt alone, with no weight updates Source
Feb 15, 2024Xuezhi Wang et al. (Google DeepMind) — Chain-of-Thought Reasoning Without Prompting — CoT paths appear just by changing the decoding Source
Experiment
Picking the next word
The cat quietly ____ on the mat
Temperature is the knob that makes the probability distribution sharper (low) or flatter (high). Low means more reproducible and more predictable; high means more variety but a bigger risk of nonsense. That's why code generation uses a low temperature. — These are illustrative values.
03
Training time and usage time — completely different cost structures
Pretraining happens once; inference happens endlessly. So in total cost, inference wins.
Pretraining is the stage where the model picks up the patterns of language itself through next-token prediction. It is the most expensive and takes the longest. The Llama 3 team's report (2024-07) records that during 54 days of training on 16K GPUs there were 466 job interruptions, 419 of them unexpected — the first document to put numbers on the fact that large-scale training is not just an algorithm problem.
Post-training is layered on top. SFT (teaching by example) → RLHF (polishing with human preferences) → RLVR (boosting reasoning by checking answers). It's far cheaper than pretraining, but this is where most of a product's character is decided.
Inference happens every time the model is used, and that flips the total-cost calculation. Chinchilla (2022) optimized training compute only and proposed about 20 tokens per parameter, but once you add the inference costs after deployment, training a smaller model for longer wins. In practice today, 1,000–2,000 tokens per parameter is common.
In 2025 one more axis appeared: spending more compute at inference time, before answering (Era 05). Now "where to spend compute" has become a three-way question.
Mar 29, 2022DeepMind (Hoffmann et al.) — Chinchilla — optimal allocation in terms of training compute Source
Dec 31, 2023Sardana, Frankle et al. — Beyond Chinchilla-Optimal — once inference demand is included, overtraining is optimal Source
Jul 31, 2024Meta (Llama 3 team) — 466 interruptions during 54 days of pretraining on 16K GPUs, 419 of them unplanned Source
Remember thisWhen you see the phrase "compute-optimal," check which cost was optimized. Depending on whether only training is counted or inference too, the right answer can be tens of times apart.
04
Model · prompt · context · tools
It's a map of where to fix things when results fall short. In practice, the answer is usually not the model but the other three.
These four parts are a diagnostic tool. When results are bad, they tell you where to fix.
Model — when the capability itself falls short. This is the most expensive and slowest fix, and in practice it is the real cause less often than you'd think.
Prompt — when instructions are vague or conflicting. For reference, in a 2026-08 revision of its docs, Anthropic changed its advice to move away from relying on role assignment and to put clear instructions and an explanation of purpose first. The "You are an expert" habit has officially been walked back.
Context — when the needed material is missing or, the other way around, buried because there is too much. The latter is more common in practice. Context is not storage capacity but an attention budget.
Tools — when you ask for something the model can't know (today's date, a company DB, the result of a calculation). Changing the model will never fix this.
In 2026, the harness is close to a catch-all name for the last three — "everything in an agent except the model."
Aug 3, 2026Anthropic — Prompt engineering docs revised — clear instructions and an explanation of purpose take priority over role assignment Source
Remember thisIf the results are bad, don't start by changing the model. Look at the prompt, the context and the tools first, in that order. Only when it's none of the three is it a model problem.
05
Next Token's 22 axes — why they were cut this way
Cut by technology name, and a box dies when its technology dies. Cut by problem, and it stays.
The criterion for an axis is "what was the problem?" Had the cut been by technology name, axis ① (prompts) should have closed around 2024 — and in fact the term prompt engineering has gone out of fashion. But the problem (how do you give instructions so you get the behavior you want?) is still alive, and papers are still coming out in 2026.
Axes can be added. When a new trend appears, it is first registered as a watch topic, and once it has gathered more than 10 records, promoting it to an axis is considered. It is not promoted automatically — a person decides.
Axes can also go quiet. If there are no events for 8 months, or the latest year falls to half or less of the year before, the system raises a question. The answer is one of four — we didn't collect / it was absorbed into another axis / it really ended / we looked and it's normal.
Each event is counted on one axis only. If it needs to be read on several axes, it is only tagged with secondary axes and not counted there. That keeps the statistics from inflating.
Figure
Records on each of 22 axes
Measured from the current records
Counts are by main axis. Even if one event spans several axes, it's counted in only one place — that keeps the total from inflating. And these numbers are observation volume, not importance.
Unit ②
How the Technology Changed, Era by Era
We split it into six eras. Each era starts from the problem the previous era left unsolved. Get that link and the rest is detail.
194320172021202320252026
011943 – 2016
The age of parts — all of today's ingredients get made
Open up a Transformer block today, and most of the parts you find already existed before 2017.
Backpropagation made multilayer neural networks trainable. It had already been described in a 1974 doctoral thesis, but a 1986 Nature paper made it widely known. That paper's key claim was not about performance. It was that hidden layers build, on their own, representations that are useful for the task — today's "representation learning" starts here.
But once networks got deep, they would not train. In 1991 the vanishing gradient was pinned down: as the number of layers and the sequence length grow, the learning signal shrinks toward 0. LSTM (1997) used gates and ResNet's residual connections (2015) used shortcuts, giving different answers to the same problem. Both are still in use today.
Translation ran into another wall. The whole sentence had to be squeezed into a single fixed-length vector, and the translation had to be made by looking at that alone. As sentences got longer, their beginnings got blurred. In 2014, attention removed this bottleneck — at every step of the output, the model chooses for itself which part of the input to look at.
The revival from the mid-2000s came not from a new theory but from conditions. Data (ImageNet, 2009), compute (GPU training, 2009–2010) and recipes (ReLU, initialization, dropout, 2010–2012) each fell into place, and AlexNet (2012) brought all three together.
Oct 9, 1986Rumelhart, Hinton, Williams — Show in Nature that, through backpropagation, hidden layers build their own representations Source
Remember thisStacking layers deep and training what you stacked are two different problems. By 1989 there was already a proof that "deep can represent it," but no guarantee that "deep can be trained." Most of the next 30 years went into closing that gap.
022017 – 2020
The Transformer and scale — dropping recurrence let models grow
Dropping recurrence made parallel training possible, and once training could run in parallel, "size" became a variable that predicts performance.
RNN-type models are sequential by design. To compute the state at step t, you need step t−1. However hard you try to parallelize training, you cannot do it along the sequence.
The Transformer replaced recurrence with self-attention. Each position looks at every other position at once, so the whole sequence is handled in a single matrix operation. The work was reshaped into the form GPUs are good at.
Then scale opened up. GPT-2 (2019) had 1.5B parameters, and OpenAI, worried about misuse, did not release the full model right away — the first case where whether to release a model became a matter of debate. The next year, GPT-3 (2020) had 175B, more than 100 times as many.
The decisive step was the scaling laws (2020-01). How much the loss drops as you increase parameters, data and compute was summed up in the form of a power law. From this moment, "bigger" stops being a gamble and becomes budget planning.
It also matters that the GPT-3 paper was titled Language Models are Few-Shot Learners. It got the model to do new tasks with nothing but examples in the prompt, without changing the weights — what we do with chatbots today was defined here.
Jun 12, 2017Google (Vaswani et al.) — Transformer — Attention Is All You Need Source
Feb 14, 2019OpenAI — Announces GPT-2 (1.5B) and decides not to release the full model over misuse concerns Source
Jan 23, 2020OpenAI (Kaplan et al.) — Scaling Laws for Neural Language Models Source
May 28, 2020OpenAI (Brown et al.) — GPT-3 (175B) — Language Models are Few-Shot Learners Source
Remember thisThe Transformer did not win because it was a "smarter design." It won because it was a design that could run in parallel on GPUs. After this period, the choice of architecture is almost always made together with hardware efficiency.
Experiment
Sequential processing (RNN) vs parallel processing (Transformer)
Thecatsatonthematquietlyatnoon
RNN0 / 9
TF0 / 9
An RNN needs step t−1 to compute the state at step t — parallelizing along the sequence is structurally impossible. Self-attention looks at every position at once, so the whole sequence becomes a single matrix operation. This picture is the whole reason model sizes exploded after 2017.
Experiment
Attention — which word is it looking at?
Cats0.00
love0.00
mats0.00
and0.00
often0.00
nap0.00
Click a word. The bars show where in the sentence that word looks, and how much. ‘nap’ is looking for who naps, so it looks hardest at ‘Cats’. Choosing for itself where to look like this is attention. Note: these are not weights taken from a real model but mock values made up to explain how it works.
032021 – 2022
The alignment era — the problem was not ability but doing what it's told
GPT-3 already existed in 2020. ChatGPT took off 2 years later not because the model got bigger but because it was made to follow instructions.
A pretrained language model is a next-token predictor. It is not something built to follow instructions. InstructGPT (2022-01) trained a reward model on data where people ranked responses, then fine-tuned the policy with that reward — this is RLHF. OpenAI made this model the API default.
That same month, Chain-of-Thought (2022-01) appeared. Instead of having the model give only the answer, having it generate intermediate reasoning steps raises accuracy on math and commonsense reasoning. It is a classic case of changing performance with the prompt alone, without touching the weights, and it is the conceptual root of today's reasoning models (Era 05).
In March came Chinchilla (2022-03). By training more than 400 models, it showed that for a given compute budget you should not just grow the parameters but grow parameters and tokens in equal proportion. It corrected the 2020 scaling laws on the allocation side.
There were two events on the image side as well. CLIP (2021-01) put images and text in the same vector space, and Stable Diffusion (2022-08) released its weights, so image generation could run on consumer GPUs. This is the practical starting point of the open-weight ecosystem.
Jan 5, 2021OpenAI — CLIP — a shared embedding for images and text Source
Jan 27, 2022OpenAI — Announces InstructGPT and makes it the default API model (RLHF) Source
Jan 28, 2022Jason Wei et al. (Google Brain) — Chain-of-Thought Prompting — generating intermediate reasoning steps Source
Mar 29, 2022DeepMind (Hoffmann et al.) — Chinchilla — parameters and tokens in equal proportion Source
Aug 22, 2022Stability AI, CompVis — Releases the Stable Diffusion v1.4 weights Source
Nov 30, 2022OpenAI — Releases ChatGPT as a research preview Source
Remember thisChatGPT was not a new model but a new interface and new post-training. This period proved that "big model = good product" is not true.
042023 – 2024
From model to system — adding search, tools and context
What decides performance started to be not one model but the system around the model.
Knowledge stored in the weights (parametric knowledge) is expensive to update and cannot point to its sources. RAG fixes both by attaching an external index — its core value is that knowledge can be updated without retraining and sources can be shown.
Tool calling replaced reliance on parsing prompts with an API contract. The model outputs a function call in a fixed JSON schema. Toolformer (2023-02) is the starting point on the research side, and computer use in 2024-10 chose screenshots and coordinate control as the last-resort general interface for software that has no API.
The context window grew by about three orders of magnitude, from GPT-3's 2K in 2020 to between 1 million and 2 million tokens with Gemini 1.5 in 2024. But putting a lot in does not mean all of it gets read — Lost in the Middle, where information in the middle gets used less, and Context Rot, where recall drops as inputs get longer, were soon reported.
The most important lesson came from failure. Autonomous agents that run on their own once given a goal collapsed under the problem of compounding reliability. With an 85% success rate per step, you get 44% over 5 steps and 20% over 10 steps. The verdict at the end of 2024 was blunt — for most real-world problems, a workflow is enough; an agent is not needed.
LLaMA 1 (2023-02) was released only to approved applicants, but its weights leaked within days, and that led to llama.cpp (2023-03), opening the era of LLMs running on laptops.
Feb 24, 2023Meta AI — Releases LLaMA 1 — later leaked, opening up the local-run ecosystem Source
Feb 15, 2024Google DeepMind — Gemini 1.5 Pro — a 1 million-token context in preview Source
Oct 22, 2024Anthropic — Claude 3.5 Sonnet computer use enters public beta Source
A lesson from failureAn 85% success rate per step looks pretty good. But over 10 steps, 0.8510 ≈ 20%. This is the most basic reason the early autonomous agents collapsed, and the reason "a loop where a person checks along the way" has become the default in practice.
052024-09 – 2025
The reasoning era — making models think longer before they answer
Next to the axis of spending compute on training, one more axis appeared: spending compute at inference time.
o1 (2024-09) opened the door. It is a model trained with RL to generate a long chain of thought before answering. From here on there are two ways to raise performance — spend more compute on training, or spend more compute on inference.
DeepSeek-R1 (2025-01) made this recipe public. The key is RLVR. Instead of a learned reward model, it gives rewards through checker functions such as matching the correct answer or running code. It worked especially well in areas like math and coding, where grading is cheap.
R1-Zero was reported to show self-checking, re-examination and strategy switching through pure RL alone, without human-made reasoning examples (SFT) — the so-called "aha moment." In the authors' framing, reasoning is not taught but incentivized. The paper was published in Nature in 2025-09.
The product form changed too. Claude 3.7 Sonnet (2025-02) introduced hybrid reasoning, which switches between instant answers and extended thinking within one model, and a thinking budget (up to 128K tokens). Intelligence became a product you adjust with a dial, rather than switch on and off.
The same year, connection standards also took shape. MCP (an open protocol for models to connect to outside data and tools) and Agent Skills (capability packages in the SKILL.md folder format) arrived, and Agent Skills was released as an open standard in 2025-12.
Sep 12, 2024OpenAI — o1-preview — Learning to Reason with LLMs Source
Jan 22, 2025DeepSeek — DeepSeek-R1 — reasoning ability emerges from pure RL Source
Feb 24, 2025Anthropic — Claude 3.7 Sonnet — hybrid reasoning, thinking budget up to 128K Source
Apr 3, 2025Anthropic — Reasoning models don't always say what they think — hints mentioned 25% of the time Source
Remember thisAs of 2026, "reasoning models" are not a separate family of models. They have been absorbed into frontier models as a mode you can switch on and off. When you see the term reasoning model in the news, it is a good idea to first check whether it means a separate product or a mode.
Experiment
Thinking budget — how inference compute relates to performance
tokens
Accuracy0%
Cost per answer×1.0
Accuracy levels off logarithmically while cost rises almost linearly. So always making it think as much as possible is not rational, and controllers that split up the budget according to how hard a problem is became a research topic of their own (2026-06 Selective Verification for Budget-Aware Reasoning). — This is a diagram of the shape of the relationship, not real measurements.
062026
The harness era — what lies outside the model decides performance
"Everything in an agent except the model" got a name — the harness.
In the first half of 2026, the word harness took hold. Birgitta Böckeler of Thoughtworks defines it as everything in an AI agent except the model — a combination of controls that act beforehand (guides) and controls that act afterward (sensors).
Mitchell Hashimoto's definition is more hands-on: every time the agent makes a mistake, you fix the system so it can never make that mistake again. Philipp Schmid's analogy helps understanding: model = CPU, context window = RAM, harness = operating system, agent = application.
Standards are being tidied up too. In its 2026-07-28 spec, MCP made the protocol core stateless. It dropped the idea of a session and the initialize handshake, and moved to a pattern (MRTR) where, if confirmation or extra input is needed, the exchange picks up again through a retry. The change suits serverless and load-balanced environments.
At the same time, what AI cannot do also became clear. ARC-AGI-3 (2026-03) is a test that asks for interactive games to be solved with no instructions, and the result was humans at 100%, frontier AI at 0.51%. Jagged intelligence — solving PhD-level problems while failing elementary-level tasks — is still there.
Open weights are close to pulling ahead. On the 2026-08-26 Artificial Analysis leaderboard, 96 of 176 were open-weight. Regulation also actually took effect — in Korea, the Enforcement Decree of the AI Basic Act came into force on 2026-07-21.
Mar 25, 2026ARC Prize — Launches ARC-AGI-3 — humans 100% vs. frontier AI 0.51% Source
Jul 21, 2026Ministry of Science and ICT — Enforcement Decree of the AI Basic Act takes effect Source
Jul 28, 2026Model Context Protocol — Spec revision — a stateless protocol core and the new MRTR pattern Source
Aug 12, 2026Stanford Digital Economy Lab — Revised Canaries in the Coal Mine — employment 19% lower for ages 22–25 in AI-exposed jobs Source
Remember thisThis era is not over yet. Even the word harness has split definitions, and what to evaluate and how is still being debated. Read it knowing that this is not settled history but history still being made.
Experiment
Records per year — Next Token's actual counts
Point at or tap a bar to see that year's count.
Show the numbers
The gray bands are the six eras above. A word of caution: this curve shows the density of records, not the speed of technical progress. The more recent the year, the more primary sources survive and the more closely things are collected, so it comes out higher. Read it with observation bias in mind.
Unit ③
The Big Turning Points of the Last 3 Years
Context · reasoning · connection standards · harnesses · open weights and regulation · and what's still unsolved.
01
2024 — Context grows by three orders of magnitude, and models start using computers
From 2K in 2020 to 2 million tokens in 2024 — about 1,000 times. But putting something in doesn't mean all of it gets read.
The context window started at 2,048 tokens with GPT-3 in 2020. GPT-4 Turbo reached 128K in 2023-11, Gemini 1.5 Pro offered 1 million in preview in 2024-02, and in 2024-06 2 million tokens opened to all developers. That's growth of about three orders of magnitude.
But results pointing the other way came out right away: Lost in the Middle (information in the middle gets used less) and Context Rot (recall drops as inputs get longer). As evaluations like Needle in a Haystack mapped recall rates by position and length, "can fit ≠ uses" became established.
So context gets redefined not as capacity but as an attention budget. This recognition leads to context engineering in 2025 — the engineering not of what to put in, but of what to leave out to stay within the budget.
Computer use (2024-10) is an expansion in a different direction. Through screenshots and coordinate-based control, it secured a last-resort universal interface for software that has no API. It's slow and fragile, but in principle its reach is unlimited.
Nov 6, 2023OpenAI — GPT-4 Turbo with 128K context Source
Feb 15, 2024Google DeepMind — Gemini 1.5 Pro — 1 million tokens in preview Source
Jun 27, 2024Google — Gemini 1.5 Pro's 2 million tokens opened to all developers Source
Oct 22, 2024Anthropic — Claude 3.5 Sonnet computer use in public beta Source
Remember this"1 million tokens" is a spec, not a guarantee of performance. The context size on a product's spec sheet and the length you can actually use are different numbers.
Figure
The growth of the context window
Log scale · tokens
Log scale. It grew by about three orders of magnitude in four years. But this is the spec, and it soon turned out that the amount actually recalled within it is much smaller.
02
2025 — The year of reasoning models
o1 and DeepSeek-R1. And the discovery that problems whose answers can be graded automatically are the key to reasoning ability.
The core recipe is RLVR. Instead of a reward model trained on human preferences, the reward comes from a checking function, such as matching against the correct answer or running code. It worked especially well in areas where grading is cheap (math and coding).
DeepSeek-R1 made this public. In R1-Zero, self-checking, re-examination and strategy switching were reported to emerge from pure RL alone, with no human-written reasoning examples (SFT). In the authors' framing, reasoning is not something you teach but something you incentivize. The paper was published in Nature in 2025-09 after review by eight referees, and the review reports were published alongside it.
The product form changed too. Claude 3.7 (2025-02) introduced hybrid reasoning, switching between instant answers and extended thinking within one model, along with a thinking budget (up to 128K). As of 2026, a 'reasoning model' is not a separate family of models but a mode you can switch on and off.
On the safety side there was bad news. The hope of monitoring intent by reading the chain of thought was shaken — Anthropic (2025-04) reported that when Claude 3.7 was given a hint, it mentioned the hint in its chain of thought only 25% of the time.
Sep 12, 2024OpenAI — o1-preview — Learning to Reason with LLMs Source
Jan 22, 2025DeepSeek — DeepSeek-R1 — reasoning ability emerges from pure RL Source
Apr 3, 2025Anthropic — Hint-mention rate of 25% for Claude 3.7 Sonnet Source
Sep 17, 2025DeepSeek — DeepSeek-R1 paper published in Nature — eight referees, review reports made public Source
Remember thisThis recipe works because these were areas where grading is cheap. For work like writing or judgment, where the right answer can't be checked automatically, the same method doesn't carry over as is.
03
2025 — A standard for connections appears
MCP and Agent Skills. Both grew out of the same premise: "context is a limited budget."
MCP is an open protocol that defines how models and agents connect to external data and tools. It is client-server, built on JSON-RPC 2.0, and has three primitives — Tools (actions the model calls), Resources (data the model reads) and Prompts (templates the user picks).
Agent Skills are capability packages in folder form that hold instructions, scripts and resources (SKILL.md). The key is progressive disclosure — only the name and description are always visible, and the actual content is loaded when needed. It was released as an open standard in 2025-12.
Why this design: if every tool definition is loaded into the context, there's no room left for the actual work. So the MCP side took the same direction with Tool Search (finding tools through search) and code execution (orchestrating tool calls in code so intermediate results don't pass through the context).
In 2026 the protocol itself was folded down. The 2026-07-28 spec made the protocol core stateless, removing the session concept and the initialize handshake. When confirmation or extra input is needed, a pattern that picks things up again through retries (MRTR) is used — a change made to fit serverless and load-balanced environments.
Risks came along too. Tool poisoning is an attack in which a malicious server hides instructions in a tool's description, changing the model's behavior before the tool is even called.
2026 — The harness: what's outside the model sets performance
"Everything in an agent except the model" got a name.
Start with definitions. Birgitta Böckeler of Thoughtworks (2026-04-02) defines the harness as everything in an AI agent except the model, and explains it as a combination of up-front controls (guides) and after-the-fact controls (sensors).
Mitchell Hashimoto's definition is more hands-on — every time an agent makes a mistake, you fix the system so it can never make that mistake again. He sees the harness not as a state but as a habit.
Philipp Schmid's analogy is the easiest way to understand it — model = CPU, context window = RAM, harness = operating system, agent = application.
Loops split into layers too. Andrew Ng's (2026-06) 3-layer loop — the agentic loop (minutes; the agent running), the developer feedback loop (hours; a person looking at the results and fixing things) and the external feedback loop (days; feedback from real users).
To sum up, this era makes one claim — even with the same model, results vary greatly depending on system design. So benchmark scores alone can't predict how a product will perform.
Jun 26, 2026Andrew Ng (The Batch) — 3-layer loop — agentic / developer feedback / external feedback Source
Aug 19, 2026OpenAI — Codex as a platform: build on the open agent harness Source
Remember thisWhen you see the word "harness," first check who is using it. It may mean a control system, a development habit or a specific product.
05
Open weights pull ahead, and regulation actually takes effect
As of 2026-08, on one leaderboard, 96 of 176 models are open-weight. But 'open' means less than you might think.
Open-weight ≠ open source. The OSAID 1.0 standard requires three things — ① the weights ② source code for the whole training and inference process ③ data information (sources, scope, labeling procedures). By this standard, Llama — and most of Qwen too — is not open source.
Licenses form a spectrum: unrestricted (Apache 2.0, MIT) → community/terms-based (Llama Community License) → revenue- or scale-threshold (a separate contract once revenue passes a certain level) → bespoke. There's a trend worth noting — the closer to the top an open-weight model is, the more it reverts to a bespoke license.
One more: open weights ≠ runnable weights. An MoE model may need little compute per token, but at inference time every expert has to stay resident in memory. Being able to download it doesn't mean you can run it.
On the regulation side, there's one distinction you must make. In the EU, entry into force (the day a law comes into existence) and application (the day obligations begin) are different. The Omnibus Regulation that entered into force on 2026-07-27 postponed high-risk obligations to 2027-12-02 and 2028-08-02. The transparency obligations for AI-generated content (Article 50), however, still apply from 2026-08-02; only generative AI systems placed on the market before that date get until 2026-12-02 to meet the marking obligation (Article 50(2)).
Jul 21, 2026Ministry of Science and ICT — Enforcement Decree of the AI Basic Act takes effect — creates a confirmation scheme for AI products and services Source
Jul 27, 2026EU — AI Omnibus Regulation (EU) 2026/1744 enters into force — high-risk obligations postponed to 2027-12-02 and 2028-08-02 Source
Aug 26, 2026Artificial Analysis — Intelligence Index — 96 of the 176 models on the leaderboard are open-weight Source
Figure
The 176 models on the leaderboard
As of Aug 26, 2026
96 of the 176 on the Artificial Analysis leaderboard (Aug 26, 2026). But open-weight ≠ open source, and license terms vary a lot from model to model.
06
Problems still open — evaluation, context, payoff
Even what to evaluate, and how, is still being debated. And there is a gap between adoption and results.
Evaluation is shaky. In 2026, evaluation awareness (a model realizing it is being evaluated) was named the biggest threat to the validity of evaluations. A model that knows it's being tested can do well only while it's being tested.
Benchmarks themselves are piling up revision histories, too. Terminal-Bench 2.1 revised 28 tasks because of changes in external dependencies, resource mismatches and specification errors, and the revised FrontierMath corrected 123 problems and removed 5 in Tiers 1–3. In other words, there's no guarantee that past and current scores can be compared.
Context rot is still here too. A 2026-05 report said frontier models miss dangerous behavior up to 30 times more often when it comes after 800,000 tokens of harmless activity.
There is a gap between adoption and results. McKinsey (2026-08) reported company-wide adoption at 44% and recognized EBIT contribution at 37%, virtually the same as the year before. Whether this is an early-adoption lag or a structural limit is not yet known.
Labor-market signals have appeared as well. The Stanford Digital Economy Lab (2026-08) reported that employment of 22- to 25-year-olds in AI-exposed occupations is 19% lower.
Mar 25, 2026ARC Prize — ARC-AGI-3 — humans 100% vs. frontier AI 0.51% Source
Jun 12, 2026Epoch AI — Revised FrontierMath — 12 problems corrected and 7 removed in Tier 4; 123 corrected and 5 removed in Tiers 1–3 Source
Aug 12, 2026Stanford Digital Economy Lab — Employment in AI-exposed occupations 19% lower for ages 22–25 Source
Aug 25, 2026McKinsey — State of AI 2026 — company-wide adoption 44%, recognized EBIT contribution 37% Source
Unit ④
Basics You Need to Know
This rearranges the “key concepts and terms” of the axis documents into learning order. Read from the top and each later term is explained by the earlier ones.
30 terms
No match. Try another word, or read the terms in context in Units ①②③.
01
Tokens and the context window
Every cost, limit and price is counted in this unit. Start here, and the rest makes sense.
Token
The smallest unit a model handles. Chunks that show up often become one token as a whole, while rare ones get split into small pieces.
Context window
The maximum number of tokens that can be processed at once, input and output combined. It grew from 2K in 2020 to 2 million in 2024.
Price per token
Input and output tokens have different prices, and output usually costs more. With prompt caching, repeated prefixes get a discount.
02
Parameters · weights · MoE
Size has split into two numbers: the total size and the size actually used.
Parameter / weight
Values adjusted through training. 7B means 7 billion of them. More isn't always better.
MoE / active parametersMixture of Experts
Only some of the experts are switched on for each token. Labels like 1T total, 32B active mean exactly that. Less computing, more knowledge.
Distillation
A small model learns to imitate a big model's output distribution. A technique for making models lighter, from 2015.
03
Pretraining · SFT · RLHF · RLVR
What matters is what each stage changes and what it can't change.
Pretrainingpre-training
The model picks up the patterns of language itself through next-token prediction. The ability to follow instructions does not come from this stage.
SFTsupervised fine-tuning
Fine-tuning on instruction–response pairs. It works well for setting format and tone.
RLHF / DPORL from Human Feedback
A reward model is trained on human preference rankings, and that reward is used to refine the policy. DPO skips the reward model and optimizes directly on preference pairs.
RLVRRL with Verifiable Rewards
Instead of a learned reward model, the reward comes from a checking function such as matching the correct answer or running code. It's especially strong in math and coding.
04
Prompt · system prompt · prompt injection
Instructions have a hierarchy, and there are ways that hierarchy breaks.
Prompt
All the input text given to a model. The shift in this era is that prompts began to be treated not as 'questions' but as task specifications.
System prompt
A layer of instructions placed above the user's input. Roles, constraints and safety rules go here.
Prompt injection
A vulnerability where untrusted input overrides the original instructions. A web page, a document or a tool description — any of them can be the way in.
Lethal trifecta
Access to private data + untrusted content + communication going outside. When all three meet, data theft becomes possible.
05
RAG · embeddings · chunking
It's the most common setup in practice, and the most common cause of failure is here too.
RAGRetrieval-Augmented Generation
Combines the knowledge in the weights with the knowledge in an external index. Its core value is updating knowledge without retraining and being able to show sources.
Embedding / vector searchembedding
Maps text to fixed-dimension vectors and searches by closeness in meaning. On the other hand, it is actually weak at searching for exact proper nouns, code and numbers.
Hybrid search / reranking
Uses dense (embedding) and sparse (BM25) search together, then re-sorts the top candidates with a cross-encoder. It's standard practice.
Chunking
Preprocessing that decides the unit of splitting. Context disappearing the moment you split is the biggest cause of RAG failure.
06
Tool calling · agents · loops
The heart of this group is why early autonomous agents fell apart.
Tool callingtool use / function calling
The model outputs a call to an external function in a set JSON schema. It replaced reliance on parsing prompts with an API contract.
ReAct loopReason + Act
A repeating cycle of Thought → Action → Observation. Since an outside observation comes in at every step, there's less hallucination and error propagation than with pure CoT.
Compounding reliability
With an 85% success rate per step, 5 steps give 44% and 10 steps give 20%. It is the most fundamental reason early autonomous agents fell apart.
ACIAgent-Computer Interface
A command set redesigned so agents find it hard to make mistakes, in place of shells and editors made for people. Even with the same model, changing only the interface makes performance differ greatly.
07
Hallucination · benchmarks · contamination
How to read scores: what they measure and what they can't.
Hallucination
Confidently generating content that isn't true. It happens structurally, because next-token prediction optimizes for 'sounding plausible.'
Contamination
When evaluation data is included in the training data. It can't be fully detected, so it's measured indirectly with new problem sets.
pass@k
A score that doesn't state k is not comparable. pass@1 and pass@10 are completely different numbers.
Evaluation awareness
A model realizing that it is being evaluated. It was named the biggest threat to the validity of evaluations in 2026.
08
Quantization · KV cache · inference cost
Why the same model runs differently on different devices.
Quantization
Representing weights with fewer bits to cut memory and bandwidth. A label like Q4_K_M tells you the method and how strong it is.
KV cache
Stores and reuses the key/value pairs of tokens already processed. It grows in proportion to sequence length and batch size, and is the main consumer of memory in LLM serving.
prefill / decode
Prefill processes the prompt all at once (compute-bound); decode generates tokens one by one (bound by memory bandwidth). They behave in opposite ways.
Falling inference costLLMflation
The drop in inference cost at equal performance — about 10-fold a year. But the total cost per workflow can actually go up — because once it gets cheaper, people use it in more complex ways.