Multimodal models: several kinds of input, one computation

A multimodal model handles more than one kind of data, such as text, images or audio. Different input routes convert those signals into representations that can interact inside a model. Output capabilities may differ from input capabilities: reading an image does not imply generating one.

A common vision-language design uses a vision encoder to represent an image and a connector to make those representations usable by a language model. Other architectures use different forms of cross-attention or shared token processing.

Pass the image, not just its filename as words

This Transformers/PyTorch example sends a local image and a question through a multimodal chat pipeline. Install transformers, torch and the model's image dependencies, and place a real image at photo.jpg. The first run downloads the model.

from transformers import pipeline

chat = pipeline("image-text-to-text", model="HuggingFaceTB/SmolVLM-256M-Instruct")
messages = [{"role": "user", "content": [
    {"type": "image", "path": "photo.jpg"},
    {"type": "text", "text": "Describe the objects and their positions."},
]}]
result = chat(text=messages, max_new_tokens=80)
answer = result[0]["generated_text"][-1]["content"]

The processor loads the image and prepares its visual representation alongside the text. A bare text prompt saying “look at photo.jpg” would not supply the image pixels to a text-only model.

The interface hides different costs

A short text message can accompany a substantial amount of visual computation. Resolution, cropping and image tiling affect how much information is represented. Audio has its own duration and encoding costs.

Combining modalities can reveal information unavailable in text alone, but it does not remove interpretation errors. Tiny labels, spatial relations and ambiguous scenes can still be difficult.

A vision-language model is one multimodal family. Its patch and token processing helps explain why a picture is not simply one extra word in the context window.