Zero-shot: a new task without demonstrations

Zero-shot usually describes performing a task without task demonstrations in the input or task-specific training examples under the stated evaluation setup. It does not mean the model learned nothing relevant during pretraining.

Instruction tuning can improve a language model's zero-shot response to a newly described task. Vision-language models offer another mechanism: compare an image with text descriptions of candidate classes.

Change the class list at runtime

This Transformers example downloads a CLIP model and classifies your local photo.jpg using descriptions chosen at inference time. Install transformers, torch and Pillow.

from PIL import Image
from transformers import pipeline

classify = pipeline("zero-shot-image-classification",
                    model="openai/clip-vit-base-patch32")
image = Image.open("photo.jpg").convert("RGB")
ranked = classify(image, candidate_labels=[
    "a photograph of a telescope",
    "a photograph of a microscope",
    "a photograph of a camera",
])

The output ranks these candidate descriptions. It does not prove that the image belongs to one of them, and changing the candidates changes the comparison.

The interesting capability is reusing learned representations for a task specified now, rather than training a new fixed classifier head for every label set. Always state what “unseen” means in a zero-shot result.