What are Embeddings? Vector Representations Explained

What are Embeddings?

In machine learning, embeddings are numerical vector representations that capture the semantic meaning of data. They transform complex, unstructured data (like text, images, or audio) into fixed-size arrays of numbers that computers can efficiently process and compare.

The key insight behind embeddings is that meaning can be represented as position in a high-dimensional space. Concepts that are similar in meaning are mapped to nearby points in this space, while different concepts are far apart.

For example, in a well-trained embedding space:

• "dog" and "puppy" would be very close together
• "dog" and "cat" would be moderately close (both are pets)
• "dog" and "quantum physics" would be very far apart

This mathematical representation of meaning is what makes modern AI applications possible — from semantic search to recommendation systems to Retrieval-Augmented Generation (RAG).

How Embeddings Work

Embeddings are created by training neural networks on large amounts of data. The training process forces the model to learn meaningful representations.

The Basic Idea

Imagine you want to create embeddings for words. You could train a neural network to predict a word from its surrounding words (or vice versa). Through this training, the network learns that words appearing in similar contexts should have similar representations.

For example:

• "The cat sat on the ___" → "mat" (and "dog" would also fit)
• "The dog sat on the ___" → "mat" (and "cat" would also fit)

Because "cat" and "dog" appear in similar contexts, they end up with similar embeddings.

From Words to Vectors

Each word (or token) is represented as a vector — a list of numbers. For example, a simple 3-dimensional embedding might look like:

• "king" → [0.8, 0.6, 0.2]
• "queen" → [0.8, 0.6, 0.9]
• "apple" → [0.1, 0.9, 0.3]

In this simplified example, "king" and "queen" are similar in two dimensions but differ in the third (perhaps representing gender). "Apple" is very different from both.

Training Process

The training process typically involves:

1. Initialize: Start with random vectors for each word
2. Predict: Use the vectors to make predictions about word context
3. Calculate Error: Compare predictions to actual data
4. Update: Adjust vectors to reduce prediction error
5. Repeat: Continue for millions of examples

Over time, the vectors converge to meaningful representations that capture semantic relationships.

Word2Vec: The Breakthrough

Word2Vec, introduced by Google researchers in 2013, was the breakthrough that made word embeddings practical and popular. It demonstrated that neural networks could learn meaningful word representations from large amounts of text.

Two Architectures

Word2Vec introduced two training approaches:

• CBOW (Continuous Bag of Words): Predicts a target word from its surrounding context words
• Skip-gram: Predicts surrounding context words from a target word

Skip-gram generally produces better embeddings for rare words, while CBOW is faster to train.

Famous Properties

Word2Vec embeddings revealed remarkable properties:

"king" - "man" + "woman" ≈ "queen"

This analogy-solving ability showed that the embeddings had captured not just word similarity, but deeper semantic relationships like gender, royalty, and tense.

Limitations

Word2Vec has important limitations:

• One vector per word: "bank" (river bank) and "bank" (financial institution) get the same vector
• Static: The embedding doesn't change based on context
• No subword information: Rare or misspelled words may have poor embeddings

These limitations led to the development of contextual embeddings.

Modern Embeddings

Modern embedding models address Word2Vec's limitations and provide much richer representations.

Contextual Embeddings

Models like BERT (2018) and its successors produce contextual embeddings — the same word gets different embeddings depending on context:

• "I deposited money in the bank" → financial institution embedding
• "We sat by the river bank" → riverbank embedding

This is achieved by using the entire sentence as input, allowing the model to consider context when generating each word's embedding.

Sentence Embeddings

Modern models like OpenAI's text-embedding-3, Cohere's embed-v3, and sentence-transformers produce embeddings for entire sentences or paragraphs. These are more useful for comparing the meaning of full texts.

Multimodal Embeddings

Some models can embed both text and images into the same vector space, enabling cross-modal search (finding images using text queries, or vice versa).

Popular Embedding Models (2026)

Understanding Dimensions

The "dimensions" of an embedding refer to the length of the vector — how many numbers are used to represent each piece of data.

What Do Dimensions Mean?

Each dimension captures some aspect of meaning. While we can't directly interpret what each dimension represents, together they encode rich semantic information:

• Lower dimensions (100-300): Capture basic semantic similarity. Faster to compute and store, but less nuanced.
• Medium dimensions (384-768): Good balance of quality and efficiency. Suitable for most applications.
• Higher dimensions (1024-3072): Capture more nuanced relationships. Better quality but more expensive to compute and store.

Choosing the Right Dimensionality

Consider these factors:

For most applications, 384-1536 dimensions provide a good balance. Use higher dimensions when quality is critical and you can afford the cost.

Measuring Similarity

The power of embeddings comes from the ability to measure how similar two pieces of data are by comparing their vectors.

Cosine Similarity

The most common similarity metric is cosine similarity, which measures the angle between two vectors:

• 1.0: Identical meaning
• 0.8-0.99: Very similar
• 0.5-0.8: Somewhat similar
• 0-0.5: Different
• Negative: Opposite meaning (rare)

Euclidean Distance

Another common metric is Euclidean distance (straight-line distance between points). Smaller distance means more similar.

Practical Example

Consider these sentences and their cosine similarities:

• "The cat sat on the mat" ↔ "A kitten rested on the rug" → ~0.92 (very similar)
• "The cat sat on the mat" ↔ "Python is a programming language" → ~0.15 (very different)

This ability to measure semantic similarity is what powers semantic search, recommendation systems, and RAG.

Applications

Embeddings are used across many AI applications.

Semantic Search

Traditional keyword search matches exact words. Semantic search uses embeddings to find results that match the meaning of a query, even if different words are used:

• Query: "how to fix a broken heart"
• Keyword match: Might miss articles about "healing after a breakup"
• Semantic match: Finds relevant content about emotional recovery

Retrieval-Augmented Generation (RAG)

RAG systems use embeddings to find relevant documents before generating answers:

1. Documents are split into chunks and embedded
2. User question is embedded
3. Most similar chunks are retrieved
4. Chunks are provided as context to the LLM
5. LLM generates an answer grounded in the retrieved information

Recommendation Systems

Embeddings can represent users and items in the same space. By finding items close to a user's embedding, you can build recommendation systems.

Clustering & Classification

Embeddings enable unsupervised clustering of similar documents, and can be used as features for classification models.

Anomaly Detection

Data points with embeddings far from the cluster center may be anomalies or outliers.

Choosing an Embedding Model

With many embedding models available, how do you choose?

Key Factors

• Quality: How well does it capture semantic meaning for your domain?
• Speed: How fast can it generate embeddings?
• Cost: What's the API pricing or compute cost?
• Dimensions: How many dimensions do you need?
• Languages: Does it support your target languages?
• Max tokens: What's the maximum input length?

Recommendations

Testing Your Choice

Always test embedding models on your specific data. Create a small evaluation set with known similarities and measure how well the model captures them. A model that works well for English text may not work well for code, medical text, or other specialized domains.