Latent space: do the work in a learned representation

A latent space holds representations of underlying factors rather than the original observations directly. An encoder can map an image to a compact latent; a decoder maps a latent back toward the image domain.

The coordinates are learned. One axis is not automatically “brightness” or “style,” and nearby points are meaningful only through the model and its training objective.

Sample through an encoder and decoder

In a variational autoencoder, the encoder can return a mean and log variance. This TensorFlow function samples a latent using the reparameterization trick, allowing gradients to pass through the learned distribution parameters.

import tensorflow as tf

def sample_reconstruction(encoder, decoder, images):
    mean, log_variance = encoder(images)
    noise = tf.random.normal(tf.shape(mean))
    latent = mean + tf.exp(0.5 * log_variance) * noise
    return decoder(latent)

The noise is independent of the encoder, while its scale and location are differentiable. Training a VAE also needs its reconstruction and distribution-regularizing objectives.

Latent diffusion uses a related systems advantage: denoise inside a learned image representation rather than performing every step over full-resolution pixels. The representation reduces work while the decoder reconstructs the visible output.