← AI Terminology

Dimensionality Reduction (PCA, t-SNE, UMAP)

Dimensionality reduction is the process of transforming high-dimensional data into a lower-dimensional representation that preserves the most important structure — making it visualisable, compressible, or better suited for downstream modelling.

PCA, t-SNE, and UMAP are the three most widely used algorithms, each with different tradeoffs.
Why It Matters in AI
High-dimensional data suffers from the curse of dimensionality: distances become meaningless, clustering fails, and visualisation is impossible. Dimensionality reduction makes data explorable (2D/3D plots), removes noisy features (PCA compression), and dramatically speeds up downstream models. Embedding spaces from LLMs are typically 768–4096 dimensions — UMAP reduces them to 2D for visualisation.
Key Points
Aspect Description
PCA Linear; preserves global variance — finds orthogonal axes of maximum variance; fast, interpretable
UMAP Non-linear; faster than t-SNE, better global structure preservation; scales to millions of points
t-SNE Non-linear; preserves local neighbourhood structure — excellent 2D cluster visualisation; slow; non-deterministic
Caveat t-SNE/UMAP visual cluster sizes and distances are not directly interpretable
Use case PCA: preprocessing/compression. t-SNE/UMAP: visualisation. Autoencoders: learned representations
Autoencoders Neural network approach — learns non-linear compression; can be much lower dim than PCA
Simple Analogy
A globe (3D) projected onto a flat map (2D) — you lose some accuracy (Greenland looks bigger than it is) but gain the ability to see everything at once and navigate easily. Different projections (PCA vs t-SNE vs UMAP) preserve different aspects of the original space, just as different map projections preserve area, angle, or distance differently.
Common Usage Examples
  • PCA(n_components=50).fit_transform(X) — reduce 768-dim BERT embeddings before clustering
  • TSNE(n_components=2, perplexity=30).fit_transform(X) — visualise class clusters in embedding space
  • umap.UMAP(n_components=2).fit_transform(X) — faster than t-SNE; scales to millions of points
  • sklearn.decomposition.TruncatedSVD — PCA equivalent for sparse matrices (TF-IDF document features)
  • Atlas (Nomic): interactive UMAP visualisation of entire Wikipedia or Common Crawl dataset embeddings
Summary
In short: Dimensionality reduction takes high-dimensional data and squeezes out the essential structure — PCA for linear compression, t-SNE/UMAP for visualising clusters in 2D.