← AI Terminology
Embedding
An embedding is a dense, low-dimensional vector representation of a discrete object (word, sentence, image, user, product) that captures semantic similarity — similar things map to nearby vectors in the embedding space.
Embeddings are the bridge between discrete real-world objects and the continuous vector operations that neural networks perform.
Embeddings are the bridge between discrete real-world objects and the continuous vector operations that neural networks perform.
Why It Matters in AI
Before embeddings, words were one-hot vectors (sparse, no notion of similarity). Word2Vec showed that "king − man + woman ≈ queen" — meaning is encoded in vector arithmetic. Today, sentence and document embeddings power semantic search, RAG, recommendation systems, and clustering. LLM token embeddings are the input representation that enables language models to process text at all.
Key Points
| Aspect | Description |
|---|---|
| Similarity | Cosine similarity or dot product between vectors → semantic distance metric |
| Applications | Semantic search, RAG retrieval, clustering, recommendations, anomaly detection, few-shot |
| Sentence/doc | BERT, sentence-transformers — full sentence → 768–1536 dim vector capturing semantic meaning |
| Word embeddings | Word2Vec, GloVe, FastText — word → 100–300 dim vector trained on co-occurrence |
| Embedding models | text-embedding-3-large (OpenAI), E5, BGE, Nomic Embed, Cohere Embed |
| Image embeddings | CLIP, ViT — image → vector in same space as text embeddings for cross-modal search |
Simple Analogy
A map coordinates system for meaning: London (51.5°N, 0.1°W) and Paris (48.9°N, 2.3°E) are geographically close; Tokyo (35.7°N, 139.7°E) is far. Embeddings create a similar coordinate system for concepts: "cat" and "kitten" are nearby; "cat" and "democracy" are far apart. Cosine similarity is the distance measurement.
Common Usage Examples
openai.embeddings.create(model="text-embedding-3-large", input="...")— 3072-dim vectorssentence_transformers.SentenceTransformer('all-MiniLM-L6-v2').encode(texts)— fast sentence embeddings- Semantic search: embed query + documents, return top-k by cosine similarity
- RAG: chunk documents → embed → store in vector DB (Pinecone, Weaviate, Chroma) → retrieve on query
- Product recommendations: embed user history + items → find nearest product embeddings
Summary
In short: Embeddings turn discrete objects into vectors where distance = similarity — the universal representation layer enabling semantic search, RAG, and every other similarity-based AI application.