← AI Terminology
VAE - Variational Autoencoder
A VAE (Variational Autoencoder) is a generative model that encodes inputs into a probability distribution over a latent space (rather than a fixed point) and learns to decode samples from that distribution back into realistic outputs — enabling both compression and generation.
Introduced by Kingma & Welling (2013); foundational to latent diffusion models.
Introduced by Kingma & Welling (2013); foundational to latent diffusion models.
Why It Matters in AI
Standard autoencoders learn a deterministic latent code — the latent space has no structure, so you cannot sample new points meaningfully. VAEs impose a prior (typically N(0, I)) on the latent distribution via the KL divergence term in the ELBO loss — the latent space becomes smooth and continuous, enabling controlled generation and interpolation. Stable Diffusion and most modern image generators operate in a VAE's compressed latent space, not pixel space.
Key Points
| Aspect | Description |
|---|---|
| VQ-VAE | Discrete latent codes via codebook — used in DALL-E 1, AudioLM, MusicGen |
| Encoder | q(z |
| KL term | Forces posterior q(z |
| ELBO loss | Reconstruction loss + KL divergence — trades reconstruction quality for latent smoothness |
| Latent diffusion | Stable Diffusion encodes images into VAE latent (64×64×4), runs diffusion there, decodes back |
| Reparameterisation | z = μ + σ·ε, ε ~ N(0,1) — makes sampling differentiable for backpropagation |
Simple Analogy
A chef who doesn't memorise a single recipe, but learns a "recipe spectrum" — any point on the spectrum is a valid, interpolatable dish. Given a dish, they can locate where it falls on the spectrum (encode). Given a spectrum position, they cook a coherent dish (decode). The magic is that nearby spectrum points produce similar, coherent dishes — not random noise.
Common Usage Examples
- Loss:
recon_loss + kl_weight * kl_divergence(mu, log_var)— ELBO objective mu, log_var = encoder(x); z = mu + torch.exp(0.5 * log_var) * torch.randn_like(mu)- Stable Diffusion:
AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")— encode/decode images - VQ-VAE:
from vector_quantize_pytorch import VectorQuantize— discrete codebook variant - Latent interpolation:
z = alpha * z1 + (1-alpha) * z2→ smooth morphing between two images
Summary
In short: VAEs learn a structured, continuous latent space by encoding inputs as distributions rather than points — enabling smooth generation, interpolation, and serving as the compression backbone of modern diffusion-based image generators.