On-Device & Local Inference
33 records · 2 notes · records tagged with this as a related axis are included.
Running it in your hand instead of sending it to a server
Run a model directly on a laptop or phone, and the cost and the lag disappear — and above all, your data never leaves the device. This axis gathers small models, techniques for shrinking models, accelerators built into devices, and records of models going into operating systems. They don't match large models in performance, but the line for "how small can it get and still be useful" is being pushed back every year.
ASUS, Dell, HP, Lenovo, Microsoft and others are taking part in a staggered rollout. The launch put AI run on the device front and center alongside creative work and gaming, but compute performance (TOPS) was not stated
In the team's own measurements, 4 MLPerf Tiny models each ran in 21.5ms or less and under 90KiB of RAM on a single Cortex-M55 core of the Alif Ensemble E8 (the anomaly detection autoencoder in 5.2KiB). The team noted the
Company-reported MTEB Code rose from 68.76 to 78.68; context is 8K (4 times its predecessor), and MRL lets the 768 dimensions shrink to 128, cutting storage by up to 6 times. Quantized, it runs on a Pixel 11 Pro in about
If agents are treated as always-on operational infrastructure, a failed update is not a problem with one feature but the loss of the means of recovery itself. A case of the principle of zero-downtime deployment coming do
A trend in which the form of contract shifts from "selling a model" to "training a dedicated model on the customer's data", with air-gapped deployment rising to a commercial requirement.
Shows that as ASR moves to LLM-based architectures, latency and customization of domain vocabulary, not just accuracy, have become competitive axes.
Shows a path by which a non-mainstream accelerator joins the mainstream inference stack through a plugin that leaves the vLLM core untouched — the barrier to hardware diversification is coming down.
A case of attacking, under an open license, the point that the bottleneck of on-device inference is kernel implementation rather than models. Per the company's report, it is 2.57 times faster than ORT WebGPU by geometric
A Saudi sovereign wealth fund affiliate puts forward an on-device AI PC as the endpoint of a sovereign AI stack — a phase in which local inference becomes an item of geopolitical procurement
The 2026 Mac mini has been reshaped into an edge node aimed at running local LLMs, and behind the rise in its base price from $599 to $899 are foundry economics in which TSMC's 2 nm wafer cost jumped 66% to $30,000 per w
The 1.22 TB/s near-memory bandwidth of Xiaomi's AI Cube came not from a cutting-edge process but from wafer-on-wafer hybrid bonding on the X-Ring O100, a chip made on an older 6 nm node. A 1.4-micrometer pitch that fuses
The local-inference deployment layer stands on its own as independent commercial infrastructure
On-device AI frameworks shifted from a closed single model to a hybrid, multi-provider platform
The local deployment stack reorganized from sole reliance on llama.cpp to MLX first + GGUF compatibility
Real measurements showing hardware-native FP4 has not yet beaten community quantization. The format race is unresolved
The stage at which an on-device runtime covers not just mobile OSes but browsers with the same stack. Figures are the vendor's own measurements
A stable-channel case of an on-device model exposed as a web platform API rather than an app. Responsibility for model distribution and storage management passes to the browser vendor
Made local inference, free of cloud lock-in and per-token billing, an official product line
On-device AI on Android evolved into a system in which a system service (AICore) distributes open models directly
GGUF absorbed a hardware-native FP4 format beyond its own K-quants
The first unified reference to move the choice of quantization from gut feel to measurement
A follow-up document to the July release. A case of an on-device family having public documentation at the level of a technical report
Material in which a consumer silicon generation change began to be promoted directly by local LLM inference performance
The point at which the path was set for GGUF to absorb vendor FP4 formats one after another
A local deployment tool expands into hybrid execution. The point at which the definition of running locally began to blur
An approach that tackles the on-device inference bottleneck with a CPU instruction-set extension (SME2) rather than a dedicated NPU. Framework integrations were announced at the same time, so it is deployed through the s
A commercial model family that treats on-device as a design goal rather than a by-product of post-hoc quantization, searching architectures with hardware in the loop. The figures are the vendor's own measurements, not in
A milestone at which small, on-device-class models began to combine multimodality with competitive benchmark scores
The 11B and 90B were the first Llama models to support image input, released alongside 1B and 3B models aimed at on-device use
The first large-scale case of OS-level on-device LLM integration
Justified the approach of small models deployable on phones through data curation
The de facto standard format for deploying local LLMs
Spread among users without dedicated hardware. Being able to run on a CPU alone was decisive
Standardized quantization as a post-processing step
Made it possible to run large models on consumer-grade GPU servers