multimodal large language model
In plain wordsAn AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type.
A large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. extended to accept, and in some cases generate, non-text modalities such as images, video, audio and documents. Most systems attach a separate encoder for each non-text modality (for example a vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence. for images) plus a small learned connector that maps its outputs into the language model's embedding space, and are then trained on paired and interleaved data; 'native' multimodal models instead tokenize all modalities and pretrain on them jointly from the start, and some decode image or audio tokens to produce media. MLLMs read charts, screenshots and handwriting, answer questions about images (visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.), converse in speech and underpin computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people. agents. Typical weaknesses include counting, fine spatial reasoning and describing objects that are not there.
We switched to a multimodal LLM so the agent can read the screenshots instead of parsing the HTML.
Origin
DeepMind's Flamingo (April 2022) showed few-shot learning over interleaved images and text, and Shaohan Huang and colleagues at Microsoft described Kosmos-1 (February 2023) as a 'Multimodal Large Language Model (MLLM)'. GPT-4 (March 2023), LLaVA (April 2023), Google's natively multimodal Gemini (December 2023) and OpenAI's GPT-4o (May 2024) made multimodality standard.
Usage note
Most frontier models are now multimodal, so 'LLM' often implicitly includes MLLMs. For models focused on images and text, vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. is the usual term.
Related
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it.
- vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
- visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
- audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.
- CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
Sources
- Huang, S., Dong, L., Wang, W. et al. (2023). Language Is Not All You Need: Aligning Perception with Language Models. arXiv:2302.14045. arxiv.org/abs/2302.14045
- Liu, H., Li, C., Wu, Q. & Lee, Y. J. (2023). Visual Instruction Tuning. NeurIPS 2023. arxiv.org/abs/2304.08485
Mentioned in
frontier modelfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules., adapteradapter, n.: A small trainable add-on slotted into a large frozen AI model to teach it a new task or style, like a plug adapter that lets one device work in many countries., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., document understandingdocument understanding, n.: Using AI to read documents such as invoices, forms, contracts and scientific papers the way a person does, taking in layout, tables and charts as well as words, to pull out the information needed., instruction-based image editinginstruction-based image editing, n.: Changing a picture by telling an AI what to change — 'make it night', 'remove the car', 'give her a red coat' — instead of using editing tools by hand., MMMUMMMU, prop. n.: A test for AI models that understand pictures as well as text: 11,500 college-level questions across 30 subjects, each built around one or more images such as a diagram, a musical score or a chemical structure., multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it., Nano BananaNano Banana, prop. n.: The nickname, later Google's official branding, for its Gemini image models, a viral hit in 2025 for editing photos from simple instructions while keeping people looking like themselves., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word., visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.