2023, in 127 pieces

Included inference cost in the objective, providing the theoretical basis for today's overtraining norm

A government document itself acknowledges that Korea has no TDM exemption like Article 30-4 of Japan's Copyright Act. The document that defines the asymmetry between Korean and Japanese law on training data

The first major training-data lawsuit brought by a news organization

A model from a Korean startup that, per Upstage, topped the Hugging Face Open LLM Leaderboard at release; the 74.2 average belonged to the instruction-tuned SOLAR-10.7B-Instruct, while the base model scored 66.04

A scalable oversight experimental paradigm asking whether a weak supervisor can steer a strong model

Automatically reuses the prefix KV cache through a radix tree

Made Sparse MoE a standard architecture of the open ecosystem and imprinted the concept of total versus active parameters

A date often wrongly cited as the passage of the AI Act

Declared an approach of pretraining all modalities together from the start, rather than attaching encoders afterward

The very fact that even ASE does not disclose quantitative results for back-end inspection AI is itself a benchmarking reference

With linear-time sequence modeling, made what comes after the Transformer a serious option

The first proposal in the prefill-decode disaggregation line

A general-purpose DeepSeek model released down to its intermediate checkpoints; DeepSeek claimed the 67B Base outperformed Llama 2 70B Base in reasoning, coding, math and Chinese comprehension

A symbol of AI-driven materials discovery, and later the subject of the biggest reproducibility dispute

Spread the frame of seeing the LLM not as a model but as an OS kernel, which became the shared metaphor for later discussions of agents, tool calling and context management

On 466 questions, humans scored 92% versus 15% for GPT-4 with plugins; shifted the focus of evaluation from hard exams to practical ability

A community test settled in as the de facto standard evaluation, establishing the recognition that the stated context length and actual recall performance differ

The context-length race entered the 200,000-token range

Made difficulty that searching cannot solve its design principle, linking it to scalable oversight research

A structured multi-agent setup that explicitly assigns roles, goals and tasks; a shift that gives up full autonomy in favor of designing workflows

The first product to put memory capacity and bandwidth, rather than compute, forward as its generational differentiator

Rephrasing or translation alone defeats n-gram decontamination, and 8–18% of HumanEval overlaps with public corpora

Agents moved from experiments into platform product strategy

The turning point at which homemade RAG pipelines began to be absorbed into built-in platform features

Its first product. Made wit and a rebellious streak explicit design goals, and presented real-time knowledge from the X platform as its differentiator

The starting point of the line of international AI safety summits, and the source of the mandate for the International AI Safety Report

Sparked the race in long-document embeddings. The first open model family to reduce the burden of chunking

The first non-binding multilateral norms for frontier AI

The first comprehensive executive action on AI at the US federal level

An early demonstration that FP8 training actually works for large models

A proposal to make measuring contamination for each benchmark a community standard

The start of a repeating cycle of chips designed to get around the rules, followed by new rules

Integrated in-flight batching, paged attention and FP8 into the vendor stack

The turning point that moved retrieval from a fixed pipeline to a decision made by the model

The starting point of shared cross-embodiment datasets

The prototype of the out-of-context memory management line (later the memory tool and others)

Later became the de facto objective function for coding-agent development

A branch that inserts an abstraction step through the prompt

A framework that shifted from writing prompts to compiling programs

A direct rebuttal of the Self-Refine/Reflexion family

A branch that gets the few-shot effect without labeling examples

The finding that initial tokens act as attention sinks; 2 years later gpt-oss adopted learnable sinks

The evolutionary prompt-optimization lineage that leads to GEPA

A truly unrestricted license and a challenger from Europe; later the base for thousands of fine-tunes

Image understanding moved from a research preview to a broadly deployed feature

Structured self-checking into 4 steps to reduce hallucinations

The first voluntary lab framework that ties safety measures to capability thresholds

An open-weight strategy of releasing benchmark, data and models as one bundle changed the landscape of the MTEB leaderboard

An attempt at a CUDA alternative aiming for systems-level performance while keeping Python syntax. Compiler competition extended to the language layer

Formalized automatic prompt optimization as a general optimization problem

The de facto standard recipe for context extension, unifying Position Interpolation and NTK-aware interpolation

HNSW became usable in an existing RDB without a dedicated vector DB, directly affecting the shape of the vector DB market

A code-specialized line trained on 500B tokens of code and code-related data on top of Llama 2; a 70B version was added on January 29, 2024

The start of Korea's first lineage of large commercial LLMs

A single-file distribution format with metadata and backward compatibility — the container image of local model distribution

Generalized ToT's tree constraint to graphs

A multi-agent approach that composes agents through conversations between them and allows human-in-the-loop

Company-reported scores: 58.7% on Codex (Instant 1.1: 52.8%) and 86.7% on GSM8K (80.9%); Anthropic said it hallucinates less and is more resistant to jailbreaks

A sign that small open-weight models were starting to overtake commercial APIs

Alibaba's first open-weight language model; the original release was trained on 2.2T tokens, raised to 2.4T in an update on September 25, 2023

An assembly-line style of multi-agent system that encodes SOPs as prompt sequences

The de facto standard format for deploying local LLMs

Transferred web-scale VLM knowledge directly to robot control, establishing the very category of VLA

A branch that applied reasoning prompting to latency optimization

In an evaluation on real, working websites, a GPT-4 agent scored 14.41% versus 78.24% for humans, nailing down the gap between demos and reality

Popularized open weights, and the starting point of the debate over whether it is open source

Halved the practical cost of long-context training. The precondition that made 32K–128K context models economically viable

A methodology that quantified faithfulness through intervention experiments

The starting point of a new frontier lab made up of people from DeepMind, OpenAI and Google. The list of founding members follows Wikipedia; an official xAI announcement has not been checked

Company-reported scores: Codex HumanEval 71.2% (Claude 1.3: 56.0%), GSM8k 88.0% (85.2%), and 76.5% on the multiple-choice section of the Bar exam (73.0%); API priced the same as Claude 1.3

The world's first regulation dedicated to generative AI

The first large-scale counterevidence to the assumption that you can simply put in a long context

Elevated oversight of superhuman AI to an explicit research program

The event that publicly confirmed benchmark numbers are not comparable

Trained a 1.3B model on 7B tokens to reach 50.6% on HumanEval, demonstrating that data quality matters more than quantity

Manages the KV cache like OS paging, cutting memory waste to under 4%; the starting point of the open-source serving standard

An improvement that made ZeRO work even on low-bandwidth clusters

Provisions on generative AI were added at this stage

Tool calling was promoted from prompt parsing to a structured JSON-schema interface; the de facto standard API for agent development

Became the basis for automated evaluation and synthetic rewards, flowing directly into post-training pipelines

Claimed that well-filtered web data alone can outperform curated corpora; MinHash fuzzy deduplication became a standard technique

Quantization that protects only the top weights as judged by activations; TinyChat runs more than 3× faster than HF FP16

Optimizes directly on preference pairs without a reward model or PPO, popularizing open-source alignment

The starting point of the MCTS-based reasoning line of work

Fine-tunes a 65B model on a single 48 GB GPU using 4-bit NF4, double quantization and paged optimizers

The reference paper for the multi-agent debate line of work

Restructured the RNN so it can be trained at Transformer level; the opening signal of a revival of linear complexity

A compromise between MHA and MQA that allows uptraining of existing checkpoints, and became the industry-standard KV-saving technique

Redefined linear CoT as a search problem

The starting gun for the long-context race; made feeding in whole documents without RAG a real option

At Google I/O, Google announced over 25 products and features powered by PaLM 2, including Bard, and said the smallest size, Gecko, can run on mobile devices even offline

The starting point of the line of work challenging CoT faithfulness

Uses a planning step to make up for zero-shot CoT's calculation and missing-step errors

Early work on the idea of textual gradients. Led to TextGrad

Shifted the focus from benchmarks with correct answers to evaluation based on human preference

A public change of position by one of the founders of deep learning. The turning point at which discussion of AI existential risk moved from the fringe into the mainstream press

A rebuttal arguing that emergence is an artifact of discontinuous metrics; with continuous metrics the curves are smooth

With three large rounds in 3 weeks, the narrative of long-term memory for LLMs peaked in the capital markets

The turning point at which large-scale training became possible with the standard framework, without DeepSpeed

A representative case of capital flowing into vector DBs at the time of the 2023 RAG boom

An open reproduction of the LLaMA training recipe that opened an era of reverse-engineering the data behind closed models

The starting point of the open VLM lineage

The event that created the category of embedded vector stores for LLM app developers

Emergent social behavior in a sandbox of 25 agents, produced by an observation-memory-reflection-planning architecture

Established at once the practice of training on synthetic and leaked conversations and the culture of LLM-as-judge evaluation

Created the story of an AI that just gets on with it once you give it a goal, while also exposing the public to infinite loops, hallucinations and runaway costs

The standard reference for iterative refinement loops without external supervision

Presented a minimal structure that runs execution, task creation and reprioritization in an endless loop

The de facto successor to CLIP in multimodal embeddings. Removing the need for very large batches greatly lowered the barrier to training

The first case of a lab putting retrieval augmentation on its own product surface; a direct demand signal for the vector DB ecosystem

The lineage of verbal reinforcement learning without weight updates. The v1 title was an autonomous agent with dynamic memory and self-reflection

Compiler-based optimization became a framework default

Better instruction understanding reduced the marginal value of prompting tricks; the axis of competition moved to models, safety and context

Released the same day as GPT-4, symbolizing that competition was moving beyond prompting tricks

Demonstrated that instruction-following can be replicated for a small sum

Dependency-free inference in pure C/C++ opened the way to running LLMs on a laptop without a GPU

Prompt engineering institutionalized as a job with a title and a salary band

Proved that controlled distribution is physically impossible; the real starting point of the open-weight ecosystem

Spread among users without dedicated hardware. Being able to run on a CPU alone was decisive

The first large attempt to distribute frontier-class weights after vetting

Wired up LoRA and Prefix/Prompt Tuning in a few lines, laying the groundwork for fine-tuning on consumer GPUs to go mainstream

Learns on its own, through self-supervised learning, when to call APIs and with what arguments — the route of building tool use into the weights

Publicly demonstrated that in a live service the system prompt does not work as a line of defense

Google said it would start with a lightweight version of LaMDA to reduce compute needs and gather feedback, before wider public availability in the coming weeks

Popularized the idea that a mismatch between the tokenizer and the model's training corpus is a real failure mode

An independent formulation of the same technique, with a 2–2.5× speedup

A one-liner that defined natural-language prompting as programming. It later became the slogan of prompt engineering discourse. The original X post was not checked directly; this relies on Quote Investigator's documentat