Intern-NCP Team presents NCP-ArchPreview with Next Concept Prediction

Presented NCP-ArchPreview, a latent-space language model that adds a Next Concept Prediction (NCP) objective — predicting discrete "concepts" that span multiple tokens — on top of next-token prediction (NTP). It builds the latent space from a product-quantized concept vocabulary derived from hidden states; a dedicated Concept Module predicts future concepts and feeds them back to the token level to guide generation, and NTP and NCP are trained jointly, end-to-end. At 8.9B parameters and 5.73T Dolma-3 tokens, it is the largest latent-space language model; the authors claim it reached OLMo-3-7B's final pretraining loss after using only 51.3% of its total training tokens, and led by 2.45 points on the final downstream macro average (5.99 points on GSM8K). Also reports uses such as lightweight domain adaptation that updates only a 17M-parameter VQ module, and injecting concept representations into the DFlash2 drafter to improve average acceptance length by 4.17%.