A model design with many "experts" inside, of which only a few work on each token.
A mixture of experts, or MoE, puts many small neural-network blocks called "experts" inside one model and uses only a few of them for each token.
A small network called the router decides which experts each token goes to. So the model can have a huge number of parameters in total while doing only part of the work at a time. A Google Brain team first showed this design really working in 2017.
In December 2023, Mistral's Mixtral 8x7B had 46.7 billion parameters but used about 12.9 billion per token, so it ran about as fast as a 12.9-billion model. The catch: nobody knows in advance which experts will be called, so all 46.7 billion must sit in memory.
Think of it like this
A snack-bar kitchen may have eight cooks, but for each order the head cook calls in just two instead of everyone crowding round. Orders go out fast even at the lunch rush. Since nobody knows who'll be called next, though, all eight have to stay in the kitchen.
For example
A name like Qwen3-Coder-480B-A35B marks an MoE model: 480 billion parameters in total, of which about 35 billion (the A stands for active) are used for each token.
Common mix-ups
Mix-upEach expert in an MoE takes one school subject, like maths or history.
ActuallyWhen the Mixtral team looked, experts were picked in almost the same way for research papers, biology and philosophy texts, with only a small difference for maths problems. Instead, the same expert was often called for pieces of text structure, such as 'self' in Python code. An "expert" is closer to a slot that shares the work than to a subject teacher.
Mix-upAn MoE is several separately built AIs glued together.
ActuallyMixtral has 8 experts in every layer and picks 2 of them afresh for each token at every layer, so a single token can pass through different experts as it moves up the layers. The experts and the router are trained at the same time, inside one model.
Try it in class
For teachers at the start of a lesson — or for students working on their own.
- Question
- Total parameters versus parameters used at once: which one sets the speed, and which one sets the memory?
- Activity
- From the Mixtral 8x7B (E0302) and DeepSeek-V3 (E0486) records, find the total parameters and the ones used per token, work out what percentage is actually working, and put it in a table. Then match "number of cooks" and "cooks called per order" from the snack-bar comparison to those two numbers, and open the 2017 starting-point record (E0070) too.
Records about this
Source-checked moments from AI history.
Established low-precision pretraining at frontier scale
Made Sparse MoE a standard architecture of the open ecosystem and imprinted the concept of total versus active parameters
Simplified MoE and became the standard reference for routing instability and load-balancing loss
The first demonstration, through conditional computation, of an architecture whose total parameters far exceed its active parameters; the ancestor of every MoE