Multimodal
73 records · 1 notes · records tagged with this as a related axis are included.
Models that read and make more than text
When one model starts handling pictures, sound, screens and even sensor signals, the places it can be used multiply. This axis gathers techniques that bind different kinds of data into a single representation, models that see and speak, and records of these landing in real products. It's also an area where evaluation gets harder as abilities grow, so any claim about how well a model does something comes with the test it was measured on.
The authors reported that, trained on 12.3 million question-answer pairs on top of Qwen3.5-4B, it ranked No. 1 among the 8 models compared, with an overall 65.64%, and was No. 1 on 7 of 12 benchmarks. They reported it wa
It shipped two versions, a 16-step distilled one and a 50-step base one, together with FP8 and NVFP4 quantized weights and a DreamBooth LoRA training script. Only a speed figure was given — the distilled version makes a
Meta claims it is the most capable US open-weights model to date. It adds safety post-training for standalone use on top of the same base model as Muse Spark 1.3, and ships with SGLang and vLLM optimizations and Helion k
Company-reported figures: an Artificial Analysis speech-to-speech index of 86.9 (GPT-Live-1 83.2, Gemini 3.8 Live 82.6), Full Duplex Bench 97%, and response latency under 1 second. xAI said it is used in Tesla vehicles a
The authors said the vision track was built from 820 videos and 2,460 questions and the track that also uses sound from 240 videos and 720 questions, all annotated by people and checked several times, and that 50 models
The GPT-6 generation reached the default chat of ChatGPT, which has more than 1.2 billion weekly users. Because it begins answering while it keeps thinking, GPT-6 Instant starts answering 44% sooner on average than GPT-5
An attempt to measure, separately from video generation quality, "whether a model has basic common sense about the physical world". Moving cognitive-development tasks into a benchmark shows world-model evaluation startin
With transcription prices down to 10–20 cents an hour, voice enters the range where all of it gets stored, like logs.
A case of a JEPA-family predictive architecture moving beyond video to molecules, clinical data and weather within the same framework. It reflects a trend toward defining world models not as "video generation" but as "do
Sells voice not as "transcribe, then process the text" but split into two tiers of model with adjustable thinking budgets — a form that separates latency and reasoning depth into products within the same interface.
Full-duplex voice came from a Chinese lab as well, in the same week as OpenAI's GPT-Live-1 (Sep 10) and Google's Gemini 3.8 Live (Sep 15) — a stretch in which the voice interface settles from one lab's differentiator int
A case of the in-context learning paradigm of LLMs being transplanted to robot manipulation, showing robot foundation models heading toward generalization without per-task fine-tuning.
A case of a large open-weight MoE coming down from general purpose to the vertical task of translation, squarely targeting commercial translation services.
Shows that a fully end-to-end unified architecture with neither a separate vision encoder nor a VAE has reached practical performance, suggesting that the standard design of unified multimodal models may change.
Shows that as ASR moves to LLM-based architectures, latency and customization of domain vocabulary, not just accuracy, have become competitive axes.
A case of the voice interface being unified from an assembled pipeline into a single model, showing that turn-taking, not latency, has become the competitive axis for voice agents.
A design that separates generative rendering from state management, pointing the way for world models to move beyond "plausible video" to playable simulators in which rules are enforced.
The point at which models for judging the authenticity of generated content moved beyond a research topic to be productized as standard components of the broadcast infrastructure stack.
An extension of the AlphaFold DB pattern of precomputing and publishing a model's outputs in full, rather than offering access to the model. The company claims early collaborators found 22% more non-coding genetic associ
A turning point at which world-model-based robot policies move from competition among closed systems to a controlled science that checks "which design elements work, and why".
A strand that claims multi-level ranking supervision makes better use of the supervision signal than contrastive learning, long the default for embedding training, and repositions the reranker not as the back end of the
AI weather models move beyond research demos to become the default forecasts in consumer products and commercial APIs
A strand that puts 3D world state into multimodal models as a native representation rather than an add-on condition. A recent case for the observation topic "world models"; since code, models and datasets were released t
A strand that evolves memory instead of putting an entire long video into the context, breaking down RAG's separation of retrieval and generation for time-series input. In the authors' own reporting, it claims the best p
A case of the diffusion language model family crossing over into image generation, adding to the accumulated evidence for the observation topic "diffusion language models". In the authors' own reporting, it claims 53.53
Instead of feeding in a long video whole, the model explores it on its own — multimodal input is folded into the agent loop
The lesson from making a 1-minute cinematic video about Yi Sun-sin's Battle of Myeongnyang is the order of steps: the prompt does not go in right away. First, the AI is made to research the topic, the Battle of Myeongnya
A strand that, instead of the ColPali lineage of reusing generative models as retrievers, returns to a single tower designed as an encoder from the start — a claim that "encoder-only" is valid again for multimodal retrie
Video generation moves from one-off clips to timeline work of extending and controlling
The first case of a major platform creating an AI persona category and, going beyond labeling, structurally treating it differently in its recommendation algorithms
Although it is an image model launched after regulatory pressure over NCII, the announcement makes no mention of watermarking or moderation
Provenance labeling such as C2PA and SynthID moves from goodwill to a regulatory compliance requirement. Dec 2, 2026 is only a transition deadline for systems already on the market, not a postponement of Article 50 as a
Brought reasoning effort and parallel tool calls to voice models and raised context from 32K to 128K. Voice became an environment for running agents
Showed in numbers that visual understanding and operating a computer are the same capability
The commercial embedding lineage officially moved past the text-only generation into a natively multimodal one
Open-weight omni-modal models caught up with the frontier, handling 10 hours of audio and 400 seconds of video
Showed that reading dense screenshots and diagrams had been the bottleneck in computer use performance
The point where the K2 line shifted from text to native multimodality, and the arrival of an open-weight agent swarm
Commercializes a video generation API in the midst of regulatory pressure over NCII. The announcement makes no mention of content safety or moderation
With an independent lab pursuing world models rather than LLMs, the landscape of the discourse split
A sign that open-source serving infrastructure, once text-only, is expanding to cover non-autoregressive and multimodal generation
Combines a Mistral-3 24B VLM with a rectified flow transformer; a release in which open weights narrowed the quality gap
Accurate infographics and multilingual text rendering redefined image generation as a reasoning task; ships with SynthID
A natively multimodal design paid off as an edge in video and document understanding
Generates editable 3D worlds from text, images and video, and exports them as Gaussian splats and meshes
97% OCR precision at compression below 10×, about 60% at 20×. A shift in thinking: images as a medium for compressing context
A likeness-rights track takes shape, separate from the copyright track
How rights are handled in generated video becomes a live industry issue
Synchronized dialogue and sound effects, better physical accuracy, and a feature for inserting yourself (cameo)
Character-consistent editing reaches mass-market product quality; carries an invisible SynthID watermark
Generates controllable environments from a prompt, staying consistent for several minutes at 720p (720 lines) and 24 fps
Generating dialogue, sound effects and background sound together with the footage ended the silent film era
Commercial embedding competition moved beyond text search to long documents and mixed-modality search
The official description runs tokens → transformer → diffusion → pixels — not purely autoregressive (AR), but a hybrid of AR token generation and a diffusion decoder
Connected web-scale semantic knowledge to robot control as an output modality
Google claimed 2.0 Flash outperforms 1.5 Pro on key benchmarks at twice the speed, launched the Multimodal Live API and Deep Research, and presented agent prototypes such as Project Astra, Project Mariner and Jules
Voice becomes a developer primitive (speech-to-speech over WebSocket)
The 11B and 90B were the first Llama models to support image input, released alongside 1B and 3B models aimed at on-device use
The lineage that replaced pipeline-based voice dialogue with a single model
Voice responses in as little as 232 ms and 320 ms on average, structurally eliminating the latency and information loss of cascaded pipelines
A case of building its own benchmark and releasing it alongside. The announcement said only that it would soon be available to early testers, so whether it was actually released to the general public has not been checked
Opus and Sonnet available on launch day in claude.ai and the Claude API, generally available in 159 countries, with Haiku to follow. Per 1M tokens: Opus $15/$75, Sonnet $3/$15, Haiku $0.25/$1.25; assessed at ASL-2 under
The starting point of the world-model lineage. The authors argue that at this scale it can be regarded as a foundation world model
Sparked the discourse that video generation models are world simulators
Declared an approach of pretraining all modalities together from the start, rather than attaching encoders afterward
Image understanding moved from a research preview to a broadly deployed feature
Commoditized multilingual ASR, making speech a standard input to LLM pipelines
Made a high-quality generative model that runs on consumer GPUs open-weight, sparking both the LoRA and ControlNet ecosystem and the copyright debate
Its Discord-based UX brought in non-developer creators on a large scale
The prototype of the few-shot VLM, connecting frozen vision and language models through cross-attention
Diffusion in latent space rather than pixel space sharply cut compute; the direct basis of Stable Diffusion
Later the text-conditioning backbone of image generation and retrieval models. The contrastive training on 400 million pairs is based on the paper arXiv:2103.00020