Cohere releases decode megakernel serving engine for North Mini Code

Released a decode megakernel serving engine for its code model North Mini Code (30B total, 3.3B active). It merges the entire decode path into a single persistent CUDA kernel containing 16 opcodes, including dense GEMM, attention and MoE routing, and adds continuous batching, paged attention, ragged sequence lengths, an OpenAI-compatible endpoint and tool calling — which the company claims makes it the first complete serving system built on megakernels. Company-reported figures: 292 tok/s on a single H100 at batch size 1 (62% of SoL), 1.58 times vLLM's 185 tok/s; average end-to-end decode throughput 1.25–1.41 times higher; and no accuracy loss up to a 256K context (SciCode 38.9% vs 38.2%).