AI Infrastructure Software

In the team's own measurements, 4 MLPerf Tiny models each ran in 21.5ms or less and under 90KiB of RAM on a single Cortex-M55 core of the Alif Ensemble E8 (the anomaly detection autoencoder in 5.2KiB). The team noted the

Disaggregated serving has come down from a research claim to an operational guide. The key is using p99 latency rather than average throughput as the yardstick, since agent loops are directly exposed to tail latency. It

A commercial implementation that moves prompt-injection defense down from model alignment to the network boundary. Starting from the premise that agents cannot be kept from being fooled, it keeps data from getting out ev

The operational foundation for agents moving from "running when a person calls them" to "waking up on their own" has become a GA cloud product. It matters that the option of giving agents their own identity came out at t

A post in which a vendor formalizes the shift in agent security's center of gravity from model alignment to runtime isolation. Its argument overlaps with AgentKernel, posted on arXiv the same week — give up on defenses a

Leaves a first-hand record that the bottleneck of AI products has moved from models to stateful infrastructure, and is a case of a coding agent actually being put to work rewriting the company's core infrastructure.

An inference-architecture shift in which, as long multi-turn and agent workloads become standard, the KV cache extends beyond GPU memory into the storage hierarchy.

Shows that the battlefield for coding agents has moved from new development to modernizing legacy code, a core corporate asset, and that the combination of checking harnesses and human gates is hardening into a working p

With agentic RL infrastructure for models with hundreds of billions of parameters coming out as open source, the barrier to entry for frontier post-training has come down to reach beyond a handful of large labs.

A strand that, instead of sending the prompt to a remote model every time, distills the specification once and sets it into a local function. In the authors' own reporting, 83.6% semantic accuracy on FuzzyBench-Hard. Pre

A case of attacking, under an open license, the point that the bottleneck of on-device inference is kernel implementation rather than models. Per the company's report, it is 2.57 times faster than ORT WebGPU by geometric

A strand that, instead of P/D disaggregation, which requires separate device pools and KV-cache transfers, overlaps P and D on the same devices by isolating only collective-communication order and state while sharing wei

A design answer to the point where demands for data sovereignty clashed with misuse monitoring. If a structure that makes safety monitoring work outside the vendor's servers becomes standard, the barrier to adopting fron

A case of a frontier open-weight model disclosing the rationale for its architecture choices ablation by ablation. In the authors' own reporting, loss and downstream accuracy do not always move together, and combining th

The language is stabilized and fully opened 3 years after its 2023 announcement. An alternative language to CUDA C++ offers a production-grade stability contract for the first time

The event in which a CUDA-alternative stack went from an independent startup to an asset of a major silicon vendor

Material showing on a standard benchmark that ROCm narrowed the gap to single-digit percentages in 1 year. A real crack in CUDA's monopoly

The point at which numerical reproducibility (determinism, tensor-hashing debugging) made the release headline. A sign that the training stack is moving from a performance race to a debuggability phase

A sign that open-source serving infrastructure, once text-only, is expanding to cover non-autoregressive and multimodal generation

A release that supports fusing communication and compute at the kernel level, changing how the communication bottleneck in large-scale distributed training and inference is tackled

The point at which the single-controller model proposed by Google's Pathways in 2022 entered the core PyTorch lineage. TorchTitan becomes an actor inside Monarch, and torchforge is built on top of Monarch

Just as Triton abstracted CUDA by one level, Helion abstracts Triton by one more. The next layer in the lineage of kernel authoring

A direction that lets users write multi-GPU kernels directly at the framework API level. On top of NCCL-style collective communication, user kernels gain one more path

The strongest positive evidence that FP4 training actually works. However, it is the vendor's own report, and no independent reproduction has been confirmed

A release that shows the move to a compiler-centered stack, with a stabilized ABI for C++ extensions and compilable control flow

An attempt to replace the checkpoint-based recovery paradigm. Treats each step as a distributed transaction

The first primary source making ROCm's maturity checkable on a standard benchmark. The debate over CUDA alternatives moved from claims to measurements

A study that tested, at the level of scaling laws, whether FP8/FP4 works. Redefined precision not as a free optimization but as a trade against capacity

The point at which a PyTorch-native training framework became established as an alternative to Megatron-LM. FSDP2, torch.compile and FP8 were bundled into a single stack

The first primary source to publish quantitative data on fault tolerance in large-scale training. Settled, for the whole industry, that checkpointing and automatic recovery are essential

A case showing that kernels must be rewritten for a specific generation of silicon. Quantitatively rebutted accuracy concerns about FP8 attention

A head-on rebuttal of the FP8 training narrative. Documented that reports that it works coexist with reports that it cannot be reproduced

An early demonstration that FP8 training actually works for large models

An attempt at a CUDA alternative aiming for systems-level performance while keeping Python syntax. Compiler competition extended to the language layer

Halved the practical cost of long-context training. The precondition that made 32K–128K context models economically viable

An improvement that made ZeRO work even on low-bandwidth clusters

The turning point at which large-scale training became possible with the standard framework, without DeepSpeed

Compiler-based optimization became a framework default

As a joint proposal by the three companies, it made FP8 the de facto standard format. A specification agreement that prevented format fragmentation

The moment a training-infrastructure team moved down into inference. The standard for serving large models in the generation before vLLM

Effectively opened up the road to long context

The orchestration layer that made PaLM-class training possible, and the execution foundation of the JAX/TPU stack

The turning point that made FP8 training a hardware-supported feature. The first case of software and silicon bound together as one

Automated the search for parallelization strategies. Redefined manual 3D parallelism tuning as a compilation problem

A redesign of FSDP1. The precondition that made FP8 training, partial freezing and compiler optimization possible

Opened kernel writing up from a handful of experts to ordinary researchers

Got around the GPU memory wall by extending into the storage tier. Established the framing that the constraint is bandwidth, not memory

The start of the lineage that lowered the hardware barrier to entry for training large models

The common foundation shared by XLA, TVM, Triton and Mojo. Ended the practice of writing a new compiler for every accelerator

The layer that made extremely large-scale training possible without rewriting code for model parallelism

Overturned research practice centered on static-graph frameworks

Removed the memory barrier to 100B-class training

The standard implementation of large-scale model parallelism

A format that removed FP16's scaling burden. Later the de facto default training precision for the TPU, Ampere and Hopper generations

The layer that made the inside of a node one big GPU. The physical precondition for tensor parallelism becoming practical

The turning point that brought distributed training down from research infrastructure to an everyday tool

The first practical compiler alternative to the monopoly of vendor kernel libraries. Brought the concept of automatic scheduling to the industry

With composable grad/jit/vmap transformations and XLA compilation, it formed the standard route for large-scale training on TPUs. The repository states that it is a research project, not an official Google product

Broke kernels down into composable parts. FlashAttention, Transformer Engine and many modern attention kernels are built on top of it

The prototype of mixed-precision training. The whole lineage of shrinking precision, FP16→BF16→FP8→FP4, is a variation on these two techniques

The starting point of the ML compiler race. The layer that made TPUs practical and the foundation of all of JAX

The standard layer that optimizes all-reduce for the hardware topology. All data-parallel, ZeRO and FSDP training runs on top of it

The first layer that spared frameworks from writing their own kernels

Lowered the barrier to general-purpose computing on GPUs — the software foundation that all of deep learning later rode on