Describes an AI that handles more than text: images, sound and video too.
A modality is a type of input, like text, images, audio or video. A multimodal AI can understand or create several of them together, such as answering a question about a photo or holding a spoken conversation.
The trick isn't giving it eyes and ears. Images are cut into small square patches and audio into tiny time slices, then lined up with text tokens so one Transformer can handle them all.
CLIP (2021) put images and text in a shared space, and Google said Gemini 1.0 (2023) was trained across modalities from the start. GPT-4o (2024) handled text, images and audio in one model; OpenAI said it replied to speech about as fast as a person.
Think of it like this
In a group chat, friends mix texts, photos and voice notes, and you follow along easily. A multimodal AI is a chat partner that reads all three in the same conversation.
For example
Snap a photo of your fridge and ask, "What can I cook with this?" It spots the ingredients and suggests a recipe.
Records about this
Source-checked moments from AI history.
Voice responses in as little as 232 ms and 320 ms on average, structurally eliminating the latency and information loss of cascaded pipelines
Declared an approach of pretraining all modalities together from the start, rather than attaching encoders afterward
Later the text-conditioning backbone of image generation and retrieval models. The contrastive training on 400 million pairs is based on the paper arXiv:2103.00020