Vision-language models: turn image regions into representations

A vision-language model (VLM) connects visual information with language. It can answer a question about a picture, describe a scene or use text to select relevant visual content. The exact route between image features and language varies by architecture.

A common starting point is to divide an image into patches. Each patch becomes a vector that a vision encoder can transform. Position information tells the encoder where those patches came from.

From pixels to patch vectors

This NumPy function shows a linear patch embedding. image has shape height × width × channels; height and width are divisible by size. The learned projection has shape size² × channels by the desired token width.

import numpy as np

def patch_tokens(image, size, projection):
    height, width, channels = image.shape
    grid = image.reshape(height // size, size, width // size, size, channels)
    patches = grid.transpose(0, 2, 1, 3, 4).reshape(-1, size * size * channels)
    return patches @ projection

The reshape and transpose preserve each patch's pixel order while putting patches into rows. Matrix multiplication then maps pixel values into learned feature coordinates. This is an input operation, not a complete visual encoder.

The language model needs a connection

Architectures such as LLaVA pass encoded visual features through a connector into the language model. Other designs use cross-attention to a visual representation. The language system can then condition its next tokens on both the question and the image features.

Higher resolution may create more patches or tiles rather than one bigger universal “image token.” That increases computation and can preserve details that would disappear when shrinking the whole image.

Reading small text or distinguishing “left of” from “right of” depends on more than naming the objects. Those tasks need useful visual representations and correct spatial interpretation. A VLM's fluent description alone does not establish either.