Eight accelerators down to two: slimming the model decides the cost

This conversation answers the conventional wisdom about the race to secure accelerators, that whoever has the most wins, by agreeing with the premise but saying that is no longer the case. The example given as evidence is vivid: a Korean-made foundation model of around 250 billion parameters was cut down from eight high-performance accelerators to two and made servable while keeping its performance, bringing costs and rental fees down to about a quarter. Added to this is the figure that the tokens used by agents grew 14-fold in half a year, from which follows the implication that operating costs scale not with how often people use the system but with the number of loops agents run on their own. Server configuration ratios are changing too: a CPU-to-accelerator ratio once put at 1 to 8 is now being discussed at 4 to 8 or even 6 to 8, and it is pointed out that the load of distributing work across many agents sits outside the accelerators. However, not even the names of techniques such as quantization or distillation come up, and neither what keeping performance means nor the size of the loss is given, so it can serve as an argument for pushing ahead but not as material for technical review.

This is a note: my own write-up after reading several sources. It has no source link; events checked against a source are kept as records.