How do photos and sounds get to stand next to words? — AI that sees and hears
Chop photos and sounds into pieces and line them up, and they can stand right where words do.
1 · Think first
The secret to AI seeing photos and hearing sounds wasn't giving it eyes and ears. Photos and sounds got cut into small pieces too, and lined up in the same row as word pieces.
Text gets cut into word pieces, photos into square tiles on a grid, and sound into very short slices of time. Once they're cut, all three are just 'pieces in a row.'
That means there's no need to build a new design. You can feed them straight into the Transformer, which lets pieces look at each other. Because it handles several senses at once, it's called multimodal.
2 · Experiment · Split it up
Try it — Split text, photos and sound into pieces
Tap the buttons to switch what goes in. If you can see that whatever you put in ends up as pieces standing in a row, you've got it. Orange marks the pieces to watch right now.
Figure
Whatever goes in takes the same road
Three kinds of input, but only one road.
3 · Quiz
Quick check
Three questions. Pick an answer and it's marked right away, with the reason why.
Today in one line
Today in one line
Chop photos and sounds into pieces and line them up,
and they can stand right where words do.
Learn more
- A 2023 document that opened up photo reading widely — and laid out what to be careful about, too — GPT-4V(ision) System Card (OpenAI, 2023)
- The 2024 announcement of a model that learned text, photos and sound together in one design — the point where spoken replies got as fast as a person's — Hello GPT-4o (OpenAI, 2024)