AI History Module · Advanced · for university students
How AI Became What It Is Today
This is a history of technology built only from events checked against primary sources. Dates, figures and sources are given as they are, and debates that are not settled are left as debates. Each item links back to the original records through its axis number.
1,295records checked
1943–2026years covered
22axes
87%from primary sources
Four units
Read in order and each later idea is explained by the earlier ones. Unit ① sets up the frame, ② sweeps through the eras, ③ narrows to the last three years, and ④ sorts out the vocabulary.
Every later unit stands on these five distinctions. It is especially important not to mix up containment and orthogonal distinctions.
01
A hierarchy of terms — three are methods, one is about output
Three form a hierarchy of methods; one describes the nature of the output. They aren't cut along the same dimension, which is why the news so often mixes them up.
The three layers are a containment of methods. Rule-based systems are AI but not ML; kernel methods and ensembles are ML but not DL. DL's defining claim is representation learning — its starting point is Rumelhart et al. (1986) showing that hidden layers build task-useful representations on their own.
Generative vs. discriminative is a difference in what is modeled — the joint distribution p(x) or p(x,y), or only the conditional p(y|x). It is an orthogonal distinction, not a level, so writing "generative AI ⊂ deep learning" is true only by accident.
LLMs blur this distinction. An autoregressive language model models P(x_t | x_<t), so formally it is a generative model, but much of its actual use is classification, extraction and judgment. LLM-as-a-judge is the classic case — a generative model used as a grader, which is why the judge model's biases carrying straight over is a standing issue on axis ⑯.
Practical implication: a comparison only works once you pin down which layer you are talking about. "We use generative AI" is a sentence that specifies nothing about the method.
Keep these apartDon't mix the containment relation (AI⊃ML⊃DL) and the orthogonal distinction (generative/discriminative) in one sentence. The moment you do, you create a question with no answer, like "is generative AI broader than deep learning?"
Experiment
What sits inside what
Press the buttons. The first three are nested inside each other; only generative AI uses a different criterion, so it overlaps them with a dashed line. That's why the question “which is broader, generative AI or deep learning?” doesn't make sense.
02
Next-token prediction — the power and limits of the explanation
A single maximum-likelihood objective over P(x_t | x_<t). A powerful explanation, but on its own it can't explain instruction following or emergence.
The objective is autoregressive maximum likelihood, i.e., minimizing cross-entropy over the corpus. Because the labels are inside the data, it is self-supervised, and this property is the structural precondition that made scaling possible.
The explanation that "every ability comes from a single objective" is powerful, but it shows its limits in two places. First, instruction following doesn't come out of it — GPT-3 before InstructGPT (2022-01) did not treat a prompt as a specification. Second, emergent abilities are not predicted by the objective — there's no way to calculate in advance which ability will appear at which scale, and there is also a counterargument that this 'emergence' is itself an artifact of how it is measured (axis ⑯).
Decoding is part of capability too. Self-Consistency (sample several paths, then take the most common answer) raises accuracy without changing the weights, and it has even been reported that CoT appears just by changing the decoding (2024-02, Google DeepMind — CoT paths emerged simply by branching on the top-k tokens instead of greedy decoding).
So "what a model can do" is not a property of the weights alone. It is the sum of objective + post-training + decoding + scaffolding. This recognition is the starting point of the 2026 harness debate (Era 06).
May 28, 2020OpenAI (Brown et al.) — GPT-3 — performs new tasks from examples in the prompt alone, with no weight updates Source
Feb 15, 2024Xuezhi Wang et al. (Google DeepMind) — Chain-of-Thought Reasoning Without Prompting — CoT paths appear just by changing the decoding Source
DebateWhether next-token prediction is "understanding" is a question Next Token does not answer. But that instruction following does not come automatically from this objective is not a matter of debate but an observed fact, and it is why alignment (axis ⑩) became a separate field of engineering.
Experiment
Picking the next word
The cat quietly ____ on the mat
The values shown are pi1/T renormalized (equivalent to scaling the softmax logits). Real serving also applies top-k and top-p truncation, so the tail is cut even more than this. — The base probabilities are illustrative values, not real model output.
03
Pretraining · post-training · inference — three cost axes
Since 2025, inference compute has also become a scaling axis. It's now a question of allocating across three terms: training, data and inference.
Training FLOPs and inference FLOPs are separate budgets, paid by different parties. Kaplan et al. (2020) and Chinchilla (2022) both optimize training FLOPs only; Beyond Chinchilla-Optimal (2023-12) showed that once inference demand goes into the objective, overtraining is optimal. The 1,000–2,000 tokens/param common today is the result of this calculation.
Bring in precision and there are even more axes. Scaling Laws for Precision (2024-11) formalized that low-precision training reduces the effective parameter count, and reported that the more training data there is, the larger the loss from post-training quantization — meaning overtraining and low-bit deployment pull against each other.
Infrastructure reliability is a real cost. Of Llama 3's 466 interruptions, 419 were unplanned, and about 78% were attributed to hardware. After these numbers came out, checkpointing and elastic training became standing topics on axis ⑱.
From 2025 on, test-time compute enters as a third axis. Epoch AI (2025-05) reported that o3 is estimated to have used 10 times the training compute of o1 — both axes are growing at once. That's why, whenever you cite "compute-optimal," you must check which cost function is meant.
Mar 29, 2022DeepMind (Hoffmann et al.) — Chinchilla — optimal allocation in terms of training compute Source
Dec 31, 2023Sardana, Frankle et al. — Beyond Chinchilla-Optimal — once inference demand is included, overtraining is optimal Source
Jul 31, 2024Meta (Llama 3 team) — 466 interruptions during 54 days of pretraining on 16K GPUs, 419 of them unplanned Source
Nov 7, 2024Tanishq Kumar et al. (Harvard) — Scaling Laws for Precision — low-precision training reduces effective parameters Source
04
Agent = Model + Harness — boundaries and the split of responsibility
The harness is "everything in an agent except the model." This boundary divides responsibility and control.
Böckeler's (2026-04-02) scheme is Agent = Model + Harness, with the harness a combination of up-front controls (guides) and after-the-fact controls (sensors). This cut line matters because it is where responsibility splits — what the model provider answers for and what the system operator answers for divide here.
de Macedo's (2026-06) formalization sets out four necessary and sufficient conditions — T1 an adaptive loop that ties together reasoning, action and observation; T2 a tool interface for perceiving and changing the environment; T3 active context management; T4 control mechanisms that do not depend on the model's compliance. T4 is the key — writing "don't do it" in a prompt is a request, not control.
One practical implication: results have repeatedly been reported in which the same model performs very differently when only the ACI (Agent-Computer Interface) is changed. It means a good part of the intelligence lies not in the weights but in the scaffolding, and it is also why benchmark scores shouldn't be read as a property of the model alone.
Caution: the boundaries between these four parts are not an agreed standard. The definition of harness alone splits at least six ways, and how it nests with context engineering is the exact opposite depending on the author (Era 06).
Aug 3, 2026Anthropic — Prompt engineering docs revised — clear instructions and an explanation of purpose take priority over role assignment Source
Keep these apartA prohibition written in a prompt is a request, not control. This is the point harness condition T4 spells out, and it is also the root reason prompt injection works.
05
The axis registry — cut by problem, not by technology
An axis is one line in a CSV, and the code isn't touched. There's no automatic promotion either — a person makes the call.
The canonical axis list is data/axes.csv. Adding an axis takes one CSV line plus one document, and no code changes. This design is what the requirement of an "evolving system" actually amounts to — if extending something requires changing code, in practice it won't get extended.
An axis's life cycle is managed with a state value — active / absorbed (+ where it was absorbed, required) / closed. A ruling on a quiet axis is recorded with its date, and only the sharp-drop flag is switched off for 90 days. Without this grace period, the same flag would have to be ruled on again every week.
The axis that counts is kept separate from the axes an event is read on.axis is one (counted by the timeline and statistics); also_axes can be several (shown only in lineages). Without this split, a single cross-cutting topic would inflate the counts on several axes at once.
This mechanism has actually caught a misjudgment. Axes ① and ② had 0 records in 2026, so they were ruled "absorbed" — but on a closer look, half of that turned out to be missed collection, and the ruling was reversed. That's why no axis is currently in the absorbed state — the value exists but isn't used.
Design principleThe tool that spots changes stops nothing. It only asks. Only the tool that checks the records' format can block publishing. Mix automatic judgment with automatic blocking, and a wrong judgment quietly becomes the standard.
Figure
Records on each of 22 axes
Measured from the current records
Counted by main axis (axis); secondary axes (also_axes) are not counted. Absolute counts depend heavily on collection history, so an axis's health is judged not by this value but by the 12-month trend within the axis.
Unit ②
How the Technology Changed, Era by Era
We split it into six eras. Each era starts from the problem the previous era left unsolved. Get that link and the rest is detail.
194320172021202320252026
011943 – 2016
Prehistory — what already existed before 2017
Practically nothing new was invented in AlexNet (2012). It took what already existed and ran it with enough data and enough GPUs.
Backpropagation was described in Werbos's (1974) thesis and popularized by Rumelhart–Hinton–Williams (1986). The universal approximation theorem from around the same time (Cybenko 1989, Hornik 1989) showed that a single hidden layer can approximate any continuous function — but it does not guarantee learnability. This distinction explains half of the 30 years that followed.
Hochreiter (1991) pinned down the vanishing gradient, and Bengio (1994) showed the same problem in RNNs. The gates of LSTM (1997) and the residual connections of ResNet (2015) are two answers to the same problem, and together with Layer Normalization (2016-07) they go into the Transformer block unchanged. The same goes for Adam (2014-12).
seq2seq (2014-09) formalized variable-length mapping, and Bahdanau attention (2014-09) removed the fixed-length vector bottleneck. At the same time, a race to get rid of recurrence ran along three lines — ByteNet (2016-10) and ConvS2S (2017-05) with dilated convolutions, and the Transformer (2017-06) with attention alone. Drop recurrence and training can run in parallel; once training runs in parallel, scale can grow. The structural premise of the scaling era came out of this race.
Organizations and hardware were also put in place during this stretch — the DeepMind acquisition (2014-01), the TensorFlow release (2015-11), the founding of OpenAI (2015-12), the PyTorch release (2016-09) and the TPU unveiling (2016-05).
Oct 9, 1986Rumelhart, Hinton, Williams — Show in Nature that, through backpropagation, hidden layers build their own representations Source
Dec 10, 2015Kaiming He et al. (MSRA) — ResNet — residual connections make very deep networks trainable Source
Keep these apartThe universal approximation theorem is a theorem about representability, not about learnability. Mix up the two claims and you misread both the stall and the revival of this period.
022017 – 2020
The Transformer and scaling laws — performance becomes a function of size
Structural parallelization opened up scale, and the scaling laws (2020-01) turned investment in scale into a predictable engineering problem.
The sequential dependence of RNNs fundamentally blocks parallelization along the sequence axis. Self-attention trades this for O(n²) computation and handles the whole sequence as a single matrix multiplication. It was the first case of hardware utilization deciding the choice of architecture, and every later discussion on axis ⑧ (Inference Infrastructure & Serving) is an extension of this trade-off.
The decision to release GPT-2 (2019-02) in stages is the starting point of the debate over release norms that axis ⑨ (Open-Weight Lineage) and axis ⑮ (Regulation, Law & Governance) deal with today.
The scaling laws of Kaplan et al. (2020-01) expressed loss as a power law in parameters, data and compute, turning investment in scale into a predictable engineering problem. This conclusion, however, was corrected 2 years later by Chinchilla (2022-03) on the allocation ratio — see Era 03 below. Reconciling the two laws is still active research in 2026 (Pearce&Song 2024-06, Skaling 2026-08).
GPT-3's framing (few-shot / one-shot / zero-shot) went on to become the standard frame for evaluating prompts, and it defined the very concept of in-context learning.
Jun 12, 2017Google (Vaswani et al.) — Transformer — Attention Is All You Need Source
Feb 14, 2019OpenAI — Announces GPT-2 (1.5B) and decides not to release the full model over misuse concerns Source
Jan 23, 2020OpenAI (Kaplan et al.) — Scaling Laws for Neural Language Models Source
May 28, 2020OpenAI (Brown et al.) — GPT-3 (175B) — Language Models are Few-Shot Learners Source
DebateSelf-attention's O(n²) quickly became a cost problem, and the 2026 frontier default is hybrid attention — a design that mixes, layer by layer, a few full-attention layers with many linear layers. In other words, "attention alone" no longer holds.
Experiment
Parallelizing the sequence axis — the real gain from removing recurrence
Thecatsatonthematquietlyatnoon
RNN0 / 9
TF0 / 9
The sequential dependence of an RNN creates a critical path that is linear in sequence length. Self-attention lowers this to O(1) depth in exchange for O(n²) computation. On GPUs the wall is the critical path rather than the amount of computation, so this trade paid off. But as n grows, O(n²) becomes a wall again, and the 2026 frontier default is hybrid attention, which mixes linear and full attention layer by layer.
Experiment
Self-attention weight distribution (illustrative mock values)
Cats0.00
love0.00
mats0.00
and0.00
often0.00
nap0.00
Select a query token and that row's attention distribution is shown. A real model has several such distributions, a different one for each head, and recomputes them at every layer. These values are not real extracted values but mock values to show the structure — the consistent conclusion of axis ⑩ (Safety, Alignment & Interpretability) is that real weights cannot be interpreted from attention maps alone.
032021 – 2022
Alignment and a corrected allocation — InstructGPT, CoT, Chinchilla, and diffusion
The gains of this period lie not in scale but in corrections to post-training (RLHF), prompting technique (CoT) and resource allocation (Chinchilla).
InstructGPT (announced 2022-01-27, arXiv 2022-03-04) formalized the pipeline of preference rankings → reward model → PPO fine-tuning, and replaced the default API model. From this point on, capability and alignment become separate engineering targets.
Chain-of-Thought (2022-01-28) was reported as a conditional result that works only above a certain scale, which soon led into the "emergent abilities" debate. This line of work has never closed — it runs on to Anthropic's faithfulness measurements in 2023-07, Oxford's CoT Is Not Explainability in 2025-07, and a rebuttal of it on 2025-12-28 (see Era 05).
Chinchilla (2022-03-29) put the compute-optimal allocation at ≈20 tokens/param. But today's practice goes far past this point, commonly using 1,000–2,000 tokens/param — because once you put inference cost, not just training cost, into the total cost, overtraining is justified (Beyond Chinchilla-Optimal, 2023-12).
The key point about the weight release of Stable Diffusion v1.4 (2022-08-22) is that its latent diffusion (LDM) architecture let it run on consumer GPUs. Being deployable is what built the ecosystem.
Jan 5, 2021OpenAI — CLIP — a shared embedding for images and text Source
Jan 27, 2022OpenAI — Announces InstructGPT and makes it the default API model (RLHF) Source
Jan 28, 2022Jason Wei et al. (Google Brain) — Chain-of-Thought Prompting — generating intermediate reasoning steps Source
Mar 29, 2022DeepMind (Hoffmann et al.) — Chinchilla — parameters and tokens in equal proportion Source
Aug 22, 2022Stability AI, CompVis — Releases the Stable Diffusion v1.4 weights Source
Nov 30, 2022OpenAI — Releases ChatGPT as a research preview Source
DebateChinchilla's 20 tokens/param is a value optimized for training compute only. Under today's conditions, where inference cost after deployment dominates the total, overtraining is rational, which is why the ratios used in practice have spread apart by a factor of tens. When you cite the phrase "compute-optimal," check which cost function is meant.
042023 – 2024
The age of systems — RAG, tool calling, long context, and autonomous agents running aground
A period that made up for the limits of parametric knowledge with retrieval and tools, and stretched context by three orders of magnitude. It is also the period when fully autonomous agents failed.
RAG combines parametric and non-parametric knowledge. Practice soon converged on hybrid retrieval (dense+BM25) + cross-encoder reranking, and the biggest cause of failure is not the retriever but chunking — the moment you split a document, context disappears.
Long context gets redefined as a problem not of capacity but of the attention budget. As Needle in a Haystack-style evaluations mapped recall by position and length, "can be put in ≠ gets used" became established. This view leads into context engineering in 2025.
The stranding of autonomous agents is a core event on axis ③. Along with compounding reliability (0.85^10 ≈ 0.20), results showed that even with the same model, changing only the ACI (Agent-Computer Interface) changes performance a great deal — grounds for the claim that intelligence also lives to a large degree in the scaffolding, not only in the weights. The 2026 harness discussion (Era 06) starts here.
One fact from the infrastructure side: the Llama 3 team's report (2024-07) records that during 54 days of pretraining on 16K GPUs there were 466 job interruptions, 419 of them unexpected. It is the first document to put numbers on the fact that large-scale training is a hardware-reliability problem.
Feb 24, 2023Meta AI — Releases LLaMA 1 — later leaked, opening up the local-run ecosystem Source
Feb 15, 2024Google DeepMind — Gemini 1.5 Pro — a 1 million-token context in preview Source
Oct 22, 2024Anthropic — Claude 3.5 Sonnet computer use enters public beta Source
Debate"Context you can put in" and "context that actually gets used" are different things. 1 million tokens is a capacity spec, not a performance guarantee, and Context Rot research was still coming out in 2026 (2026-05 Classifier Context Rot: after 800,000 tokens of harmless activity, risky behavior was missed up to 30 times more often).
052024-09 – 2025
Test-time compute — the second axis of scaling, and RLVR
The second axis of the scaling laws opened up. And verifiable rewards (RLVR) became the standard recipe for pushing up reasoning ability.
Test-time compute is the second axis of scaling. It raises performance by increasing the number of thinking tokens, the number of samples and the breadth of search. Epoch AI (2025-05) estimated that o3 used 10 times the training compute of o1 — meaning both axes are growing at once.
RLVR and GRPO are the core recipe. GRPO computes advantages from the relative ranking of several samples for the same prompt, without a separate value function (critic), which lowers memory and compute costs. The DeepSeek-R1 paper went through review by 8 referees and was published in Nature (2025-09-17), with the review reports released alongside it.
At the same time, the safety asset of being able to read the CoT to monitor intent was shaken. Anthropic (2025-04-03) reported that Claude 3.7 Sonnet mentioned a hint it had been given in its chain of thought only 25% of the time. An Oxford group (2025-07) pointed out that about 25% of 1,000 recent CoT papers treat CoT as an interpretability technique without justification, and UNC (2025-12-28) rebutted this. This debate has not closed — in 2026-02, mechanistic evidence was also reported for faithfulness dropping at the 70–85% point of the chain's length.
On the standardization axis: MCP, a JSON-RPC 2.0-based client-server protocol, defined three primitives — Tools/Resources/Prompts — and Agent Skills implemented progressive disclosure (bringing into the context only what is needed, only when it is needed) as a folder convention. Both share the premise that "context is a finite budget."
Sep 12, 2024OpenAI — o1-preview — Learning to Reason with LLMs Source
Jan 22, 2025DeepSeek — DeepSeek-R1 — reasoning ability emerges from pure RL Source
Feb 24, 2025Anthropic — Claude 3.7 Sonnet — hybrid reasoning, thinking budget up to 128K Source
Apr 3, 2025Anthropic — Reasoning models don't always say what they think — hints mentioned 25% of the time Source
DebateThe ability to monitor chains of thought has been called "a new and fragile opportunity," because RL pressure can destroy CoT faithfulness. The faithfulness debate (2023-07 → 2025-04 → 2025-07 → 2025-12 → 2026-02) is still ongoing, and which side wins will change the design premises of the whole system of AI oversight.
Experiment
Test-time compute — the shape of diminishing returns
tokens
Accuracy0%
Cost per answer×1.0
All this diagram shows is the shape (accuracy saturating logarithmically, cost linear). The real coefficients vary widely by task, model and how checkable the answers are. In domains where checking is cheap (math, coding), combined with RLVR, the slope is steep; where checking is expensive, it is much gentler. That Claude 3.7 Sonnet's thinking budget tops out at 128K tokens is a spec, not an optimum.
062026
Harness engineering — a term split apart, and questions still open
The same word is used in at least six different ways. This ambiguity itself is the state of things as of 2026.
Harness is the word whose definition splits the most. There are at least six strands — a control system from the user's side (Böckeler), a practice norm (Hashimoto), an organizational operating model (OpenAI 2026-02), a product runtime (OpenAI 2026-08 Codex), an OS analogy (Schmid), and eight components of enterprise infrastructure (Databricks). An academic formalization (de Macedo 2026-06) proposed four necessary and sufficient conditions while also criticizing that this ambiguity gets in the way of comparing systems and of publicly crediting designs.
Even the question of which contains which gets opposite answers. Böckeler sees harness engineering as a form of context engineering, while Addy Osmani says the reverse: context engineering is one component within harness engineering. They cannot be forced into one view — the direction flips depending on where a practitioner started from.
Shaky evaluation reliability is another mark of this period. Evaluation awareness (a model recognizing that it is being evaluated) was named the biggest confounder of evaluation validity in 2026, and benchmarks themselves have growing revision histories (28 tasks revised in Terminal-Bench 2.1; in the revised FrontierMath, 123 problems revised and 5 removed in Tiers 1–3 alone). On 2026-08-27, a piece even appeared that directly criticized the sample quality of LMArena.
Real-world impact also began to be measured. The Stanford Digital Economy Lab (2026-08-12) reported that employment in AI-exposed jobs is 19% lower for people aged 22–25, and McKinsey (2026-08-25) reported company-wide adoption at 44% and recognized EBIT contribution at 37% — practically the same as the year before. The gap between adoption and results is this period's open question.
Mar 25, 2026ARC Prize — Launches ARC-AGI-3 — humans 100% vs. frontier AI 0.51% Source
Jul 21, 2026Ministry of Science and ICT — Enforcement Decree of the AI Basic Act takes effect Source
Jul 28, 2026Model Context Protocol — Spec revision — a stateless protocol core and the new MRTR pattern Source
Aug 12, 2026Stanford Digital Economy Lab — Revised Canaries in the Coal Mine — employment 19% lower for ages 22–25 in AI-exposed jobs Source
Aug 26, 2026Artificial Analysis — 96 of the 176 models on the leaderboard are open-weight Source
Open questions① Will the many meanings of harness converge, or harden into separate meanings at separate layers? ② Under evaluation awareness, what can we say a benchmark measures? ③ Is the gap between adoption (44%) and EBIT contribution (37%) an early-adoption lag or a structural limit? Next Token does not answer these questions. It just keeps piling up observations.
Experiment
Events per year (n=1,295, including those checked against primary sources)
Point at or tap a bar to see that year's count.
Show the numbers
The gray bands mark the eras. This distribution shows observation density, not event density — it carries a strong recency bias, and the prehistory (1943–2016) comes out low because only items that have already been canonized remain. This is why Next Token runs drop and surge detection per axis as a separate tool: you have to look at trend changes within an axis, not absolute counts.
Unit ③
The Big Turning Points of the Last 3 Years
Six threads. The last section leaves the questions Next Token doesn't answer as they are.
01
2024 — Long context and computer use, and 'can fit ≠ uses'
Capacity grew by three orders of magnitude, but effective recall didn't keep up. It was the year context was redefined as an attention budget.
The expansion came through both architecture and recipes. RoPE-based YaRN (interpolation that differs by frequency band + attention temperature correction) became the standard extension recipe, while KV-cache compression (MLA) and hybrid and sliding-window attention took care of the cost side.
But effective recall didn't keep pace with capacity, and the problem is still live in 2026. Classifier Context Rot (2026-05) reported that frontier models miss dangerous behavior up to 30 times more often when it comes after 800,000 tokens of harmless activity — a big deal, because it means safety monitoring is vulnerable to context length. In 2026-06, a pattern of giving up early, before the context window is used up, was also reported.
The economics moved too. As prompt caching reused the KV of repeated prefixes and passed the discount on into API prices, whether a long-context strategy made sense began to be decided by the price sheet.
Computer use hinges on grounding (anchoring language to on-screen coordinates), and the OSWorld family is its test bench. In 2026-04, GPT-5.5 reported 78.7% on OSWorld-Verified — a big jump in two years, but still short of full automation.
Nov 6, 2023OpenAI — GPT-4 Turbo with 128K context Source
Feb 15, 2024Google DeepMind — Gemini 1.5 Pro — 1 million tokens in preview Source
Jun 27, 2024Google — Gemini 1.5 Pro's 2 million tokens opened to all developers Source
Oct 22, 2024Anthropic — Claude 3.5 Sonnet computer use in public beta Source
May 12, 2026Sam Martin, Fabien Roger — Classifier Context Rot — dangerous behavior after 800,000 tokens missed up to 30 times more often Source
DebateWhether long context would replace RAG was a hot question in 2024, and the 2026 answer is close to it didn't. Recall degradation, cost and latency all remain, and standard practice is still retrieval plus curated context.
Figure
The growth of the context window
Log scale · tokens
Log scale. The expansion itself was supported by YaRN-style recipes and KV compression, but effective recall didn't keep up. Citing the spec size as a performance metric is the most common misreading in this period.
02
2025 — How RLVR became the standard recipe
RLVR and GRPO. And the year the monitorability of chains of thought was shaken.
GRPO was the key on the cost side. Without a separate value function (critic), it computes the advantage from the relative ranking of several samples for the same prompt, cutting memory and compute. This simplification is what made reproduction possible for the open camp.
The limits of RLVR are the limits of this recipe. The slope is steep only where the verifier is cheap. In areas where checking is expensive or impossible (writing, strategic judgment, interpersonal interaction), the same method doesn't work, and reward hacking shows up right away.
The faithfulness debate continues into 2026. 2023-07 Anthropic's measurement → 2025-04 hint mentions at 25% → 2025-07 Oxford's CoT Is Not Explainability (pointing out that about 25% of 1,000 recent CoT papers treat CoT as an interpretability technique) → 2025-12 a rebuttal from UNC → 2026-02 mechanistic evidence of faithfulness dropping at the 70–85% point of chain length. It is not closed.
Why this matters: the monitorability of chains of thought is one of the few assets today's AI oversight relies on, and RL pressure can destroy it. That's why it has been called "a new and fragile opportunity." Which side wins here changes the design assumptions of all of axis ⑩.
Sep 12, 2024OpenAI — o1-preview — Learning to Reason with LLMs Source
Jan 22, 2025DeepSeek — DeepSeek-R1 — reasoning ability emerges from pure RL Source
Apr 3, 2025Anthropic — Hint-mention rate of 25% for Claude 3.7 Sonnet Source
Sep 17, 2025DeepSeek — DeepSeek-R1 paper published in Nature — eight referees, review reports made public Source
DebateAttempts to reproduce R1-Zero's 'emergence' claim have split. Whether it comes from pure RL alone, or reasoning patterns already present in the pretraining corpus were incentivized to the surface, can't be cleanly separated by experiment. Next Token records the authors' claims and the review process, but does not rule on them.
03
2025–2026 — How the MCP spec changed, and progressive disclosure
The changes leading up to going stateless (2026-07) were a process of the protocol folding itself to fit deployment environments.
How the transport layer changed: HTTP+SSE → Streamable HTTP (2025-03-26) → stateless core + MRTR (2026-07-28). The direction is consistent — dropping long-lived connections and server-side session state. The realities of deployment (serverless, autoscaling, load balancers) shaped the protocol.
The other primitives deserve a look too. Elicitation (the server asks the user a follow-up question mid-run) and Sampling (the MCP server, in reverse, asks the client's LLM to run an inference — borrowing intelligence without a model key of its own) make the client-server relationship run in both directions.
Vertical vs. horizontal: if MCP is the vertical axis of agent↔tool, A2A is the horizontal axis of agent↔agent, where agents discover one another through an Agent Card. Governance moved in 2025-12 to the AAIF under the Linux Foundation, with MCP, goose and AGENTS.md as its founding projects.
Supply-chain security is the real work of 2026. Skill Card (an open trust artifact holding the publisher's identity, capabilities and scan results) and SkillSpector (detecting hidden instructions with static analysis + LLM semantic analysis) came out in 2026-06. That's because the fact that a tool description is itself a prompt has become an attack surface.
CautionA tool's description is a prompt the model reads. So connecting a third-party MCP server is the same as attaching untrusted text right next to the system prompt. Plugging in servers from a registry at random without realizing this amounts to building a prompt-injection path yourself.
04
2026 — A map of the many meanings of "harness"
The same word is used in at least six different ways. Even which contains which is the exact opposite depending on the author.
Listing the six: a user-side control system (Böckeler), a working discipline (Hashimoto), an organizational operating model (OpenAI 2026-02 — humans steer and agents execute), a product runtime (OpenAI 2026-08, Codex CLI+SDK+app-server+MCP = "open agent harness"), an OS analogy (Schmid), and eight elements of enterprise infrastructure (Databricks — system prompt, tools, sandbox, file system, memory, feedback loop, guardrails, observability/governance).
On what contains what, the views directly contradict each other. Böckeler and HumanLayer see harness engineering as a subset of context engineering, while Addy Osmani, the other way around, sees context engineering as a component inside harness engineering. They can't be forced into one — the direction flips depending on whether a practitioner expanded outward from context management or broke things down from system design.
The academic formalization (de Macedo 2026-06-08, arXiv:2606.10106) sets out conditions T1–T4 while also criticizing that this ambiguity gets in the way of comparing systems and crediting design contributions. In other words, the attempt to formalize is itself evidence that the problem exists.
Next Token's position: don't write about what hasn't converged as if it had. Each definition is recorded with its speaker and date, and no ruling is made on which one is right.
Jun 26, 2026Andrew Ng (The Batch) — 3-layer loop — agentic / developer feedback / external feedback Source
Aug 19, 2026OpenAI — Codex as a platform: build on the open agent harness Source
05
What 'open' really means under OSAID, and how effective dates are structured
The closer to the top an open-weight model is, the more it tends to revert to a bespoke license. And entry into force and application are different dates.
A change in a leaderboard's makeup reflects both a reversal in capability and a change in the sample's composition. According to Artificial Analysis (2026-08-26), the top of the Intelligence Index is Claude Opus 5 at 63 vs. Kimi K3 at 60, and 96 of all 176 models are open-weight. What to watch is not the absolute size of the gap but how fast the gap has closed.
In architecture vocabulary, too, there are phases where the open camp leads — MLA, Sparse Attention, MTP, Gated DeltaNet, Hybrid Mamba-Transformer. Hybrid attention, the default at the 2026 frontier, is the leading example.
The practical implication for regulation is layers of dates. Mix up passage / promulgation / entry into force / phased application, and your preparation timing goes completely off. That's why Next Token never mixes these four in regulation entries, and it's the same reason litigation entries always state the stage of proceedings (a summary judgment is not a final judgment).
Voluntary norms run alongside. Companies have begun signing the code of practice for EU AI Act Article 50 (2026-07, Cohere), and provenance standards like C2PA and SynthID are its technical flip side.
Jul 21, 2026Ministry of Science and ICT — Enforcement Decree of the AI Basic Act takes effect — creates a confirmation scheme for AI products and services Source
Jul 27, 2026EU — AI Omnibus Regulation (EU) 2026/1744 enters into force — high-risk obligations postponed to 2027-12-02 and 2028-08-02 Source
Aug 26, 2026Artificial Analysis — Intelligence Index — 96 of the 176 models on the leaderboard are open-weight Source
Keep these apart① open-weight vs. open source ② downloadable vs. runnable ③ date of entry into force vs. date of application ④ summary judgment vs. final judgment. All four are the distinctions most often blurred in news articles.
Figure
The 176 models on the leaderboard
As of Aug 26, 2026
This is a sample defined by leaderboard listing, not market share. Even so, the direction of the shift is clear, and the opposite trend is observed at the same time: the closer to the top, the more models return to proprietary licenses.
06
Open issues, and how observation bias is handled
Under evaluation awareness, what can a benchmark be said to measure?
The problem of evaluation reliability has three layers — ① data contamination (can't be fully detected; measured indirectly with new problem sets), ② evaluation awareness, ③ grader bias (LLM-as-a-judge passes the judge model's biases straight through). On 2026-08-27, a piece even came out directly criticizing the quality of LMArena's samples (in its own sample of 500 votes, it disagreed with 52%).
Next Token has biases of its own. The number of events per year reflects observation density, not event density — the more recent the period, the more primary sources survive and the denser the collection, while prehistory keeps only the items that have already become canon, so it comes out low. That's why a separate tool looks not at absolute counts but at changes in the trend within each axis.
Three open questions are left as they are. ① Will the many meanings of harness converge, or harden into separate meanings by layer? ② Under evaluation awareness, what can a benchmark be said to measure? ③ Is the gap between adoption (44%) and EBIT contribution (37%) an early-adoption lag or a structural limit?
Next Token does not answer these questions. It just keeps piling up observations. Writing as if the answers were in is the failure this project most wants to avoid.
Mar 25, 2026ARC Prize — ARC-AGI-3 — humans 100% vs. frontier AI 0.51% Source
Jun 12, 2026Epoch AI — Revised FrontierMath — 12 problems corrected and 7 removed in Tier 4; 123 corrected and 5 removed in Tiers 1–3 Source
Aug 12, 2026Stanford Digital Economy Lab — Employment in AI-exposed occupations 19% lower for ages 22–25 Source
Aug 25, 2026McKinsey — State of AI 2026 — company-wide adoption 44%, recognized EBIT contribution 37% Source
Open questionsNext Token does not answer the three questions above. Instead of rulings it piles up observations, and when something is overturned it doesn't erase it but only changes its status. That's the only way to keep this material readable later on.
Unit ④
Basics You Need to Know
Each entry takes the term definitions from the axis documents, rearranges them into learning order, and rewrites them to put first the distinctions most often blurred in practice.
30 terms
No match. Try another word, or read the terms in context in Units ①②③.
01
Tokens and the context window
Tokenization bears on performance, cost and fairness across languages all at once. It is not neutral preprocessing.
Token
A subword unit. The BPE family is standard, learning merge rules from corpus statistics. Strings not in the vocabulary are broken down as far as single bytes.
Context window
A spec-sheet ceiling, not the same as effective recall. Lost in the Middle / Context Rot is that gap.
Price per token
The fall in inference cost at equal performance — about 10-fold a year — is called LLMflation. But the total cost per workflow can actually go up (the inference paradox).
02
Parameters · weights · MoE
Total parameters set the VRAM requirement; active parameters set compute per token. Mix the two up and comparisons become meaningless.
Parameter / weight
A proxy for knowledge capacity. But data quantity and quality, and post-training, make a big difference in performance at the same size.
MoE / active parametersMixture of Experts
Total ≠ active. At inference time every expert must stay resident in memory, so open weights ≠ runnable weights. All-to-all communication and load imbalance are the hard problems in serving.
Distillation
Since 2025 it has become a geopolitical issue — training a competing model on another model's outputs runs into both terms of service and export controls.
03
Pretraining · SFT · RLHF · RLVR
Post-training decides most of a product's character. Capability and alignment are separate engineering problems.
Pretrainingpre-training
Self-supervised maximum-likelihood training. Data curation (deduplication, model-based filtering) makes a big difference in performance, and the limited supply of high-quality public text gave rise to the 'peak data' debate.
SFTsupervised fine-tuning
The starting point was instruction tuning (the FLAN line) boosting zero-shot generalization. Data quality matters more than quantity.
RLHF / DPORL from Human Feedback
In RLAIF, instead of people, the AI critiques and revises itself according to stated principles (Constitutional AI). It goes hand in hand with the trend of putting behavioral norms into natural-language documents, like the Model Spec.
RLVRRL with Verifiable Rewards
Combined with GRPO, it became the standard recipe for 2025's reasoning models. In areas where checking is expensive, the same method doesn't work, and reward hacking shows up right away.
04
Prompt · system prompt · prompt injection
A prohibition written in a prompt is a request, not control. Every injection attack starts from here.
Prompt
The medium of in-context learning. The few/one/zero-shot distinction is about how many examples are in the prompt — an evaluation frame established by the GPT-3 paper.
System prompt
The instruction hierarchy is a learned preference, not an enforced priority. So the hierarchy holds only probabilistically.
Prompt injection
It splits into direct (the user does it) and indirect (hidden in content the model reads). In the MCP ecosystem, tool poisoning (hiding instructions in tool descriptions) and line jumping (changing behavior before any call) form a separate family of attacks.
Lethal trifecta
Willison's (2025) formulation. It serves as a practical checklist for designing agent permissions — cutting one of the three is the basic form of defense.
05
RAG · embeddings · chunking
The biggest cause of failure is not the retriever but chunking.
RAGRetrieval-Augmented Generation
A combination of parametric and non-parametric knowledge. The 2024 prediction that long context would replace it had not come true as of 2026.
Embedding / vector searchembedding
ANN (HNSW, IVF, PQ) gains speed and gives up accuracy. MTEB's core conclusion is that no single embedding dominates every task.
Hybrid search / reranking
Cross-encoder reranking feeds the query–document pair in as a single input, gaining accuracy at the cost of latency. Late interaction (the ColBERT family) aims for the middle ground.
Chunking
Context-preserving chunk embeddings (like voyage-context-4 in 2026-06, which handles documents over 32K without chunking) and GraphRAG (building a knowledge graph in advance) are responses heading in different directions.
06
Tool calling · agents · loops
Compounding reliability and the ACI. And the four harness conditions.
Tool callingtool use / function calling
The schema is the interface contract. Because loading every tool definition into the context is costly, workarounds such as Tool Search and code execution appeared.
ReAct loopReason + Act
The prototype of today's agent loops. It is the minimal implementation of harness condition T1 (an adaptive loop tying together reasoning, action and observation).
Compounding reliability
This arithmetic made human-in-the-loop the de facto default, and led to the late-2024 conclusion that 'most practical problems need only a workflow, not an agent.'
ACIAgent-Computer Interface
Empirical support for the claim that a good part of intelligence lives not just in the weights but in the scaffolding as well. It's why benchmark scores shouldn't be read as a property of the model alone.
07
Hallucination · benchmarks · contamination
In 2026, the validity of evaluation is shaky on three layers at once.
Hallucination
RAG makes answers easier to check by attaching sources, but it doesn't get rid of hallucination. How a model behaves when retrieved documents conflict with its parametric knowledge is a research topic of its own (contextual faithfulness vs. parametric faithfulness).
Contamination
A holdout (a private subset that only the creators have) is the standard response. But even a holdout slowly leaks out through repeated submissions.
pass@k
It should be read together with execution-based checking (grading by actual run results, not the output string). In coding benchmarks, figures that don't state both are hard to use as evidence.
Evaluation awareness
Combined with sandbagging (deliberately hiding capabilities), it can neutralize safety evaluations entirely. One direction under discussion is to make evaluation and deployment environments impossible to tell apart, as a design goal.
08
Quantization · KV cache · inference cost
Serving bottlenecks are mostly about memory, not compute.
Quantization
bpw (bits per weight), imatrix (adjusting strength layer by layer to reflect activation distributions), QAT (simulating quantization during training). NVFP4/MXFP4 get hardware acceleration, but quality is a separate matter.
KV cache
PagedAttention (non-contiguous allocation in fixed-size blocks), RadixAttention (sharing across requests with a prefix tree) and MLA (compression into low-dimensional latent vectors) each attack this bottleneck at a different layer.
prefill / decode
Because they behave in opposite ways, this is the rationale for separating them (PD disaggregation). The SLO axes are TTFT (time to first token) and TPOT (time per token), and the metric is not throughput but goodput (counted by requests that meet the SLO).
Falling inference costLLMflation
The inference paradox (Inference Paradox). You need to track 'price-at-fixed-capability' (the price of the cheapest model that reaches a given benchmark score) over time, and keep it separate from price changes for individual models.