Diffusion models: learn how to remove noise

A diffusion model learns to turn noisy data into structured samples. In a standard denoising diffusion setup, training adds known amounts of noise to real examples. A neural network learns a prediction that helps reverse that corruption, such as the noise that was added.

Generation starts from random noise and repeatedly applies the learned model with a sampling schedule. The image emerges through successive updates. This differs from autoregressive text generation, which commits to successive tokens.

The model predicts; the scheduler updates

Diffusers separates the neural network from the scheduler. This function exposes their repeated interaction for a pretrained DDPMPipeline, such as google/ddpm-cifar10-32. It returns an image tensor in the range zero to one. Install torch and diffusers; loading the pipeline downloads its weights.

import torch

def denoise(pipe):
    device = pipe.unet.device
    size = pipe.unet.config.sample_size
    channels = pipe.unet.config.in_channels
    sample = torch.randn(1, channels, size, size, device=device,
                         dtype=pipe.unet.dtype)
    pipe.scheduler.set_timesteps(pipe.scheduler.config.num_train_timesteps)

    with torch.inference_mode():
        for timestep in pipe.scheduler.timesteps:
            prediction = pipe.unet(sample, timestep).sample
            sample = pipe.scheduler.step(prediction, timestep, sample).prev_sample

    return (sample.clamp(-1, 1) + 1) / 2

The same network weights are used at every step, with the timestep telling the network which noise level it is handling. Here the scheduler keeps its configured DDPM sampling process. More steps mean more network evaluations; other compatible samplers can trade evaluations against output quality differently.

Pixels or a smaller representation

Pixel-space diffusion works directly on image arrays. Latent diffusion instead performs the expensive repeated process in a compressed representation, then uses a decoder to produce pixels. Text conditioning can guide the network toward a requested scene.

The loop above is unconditional pixel-space generation: it has neither a text encoder nor a latent decoder. Those components belong to other pipelines. The central relationship remains the same: a learned prediction guides a scheduled update of the current sample.