Your documents are now split into clean DocumentChunk objects, but the machine still has no idea that "cats and dogs" are close while "cats and quantum mechanics" are worlds apart. This step turns text into coordinates, stores them in a vector database, and retrieves nearest neighbors by distance.
30-Second Version · Remember Just These 3 RulesAll lab numbers are either cited or marked illustrative
01Cosine similarity measures direction only, never vector length.
02Skip MTEB overall averages—check the Retrieval split first, then benchmark on your own queries.
03Low scores? Inspect your chunks first. Setting limit without a threshold feeds noise straight to the LLM.
What Embeddings Actually Do: Encoding Meaning as Coordinates
An embedding model does not generate text; it outputs a fixed-length numeric array—a point in semantic space. Retrieval is simply finding the points closest to the query.
The vector dimensionality is fixed by the model: OpenAI text-embedding-3-small outputs 1,536 dimensions, text-embedding-3-large outputs 3,072, while Alibaba Cloud Model Studio's text-embedding-v3 and open-source BGE-M3 default to 1,024 dimensions.
For text retrieval, "closest" usually means highest cosine similarity, ranging theoretically from -1 to 1. Drag the orange arrow in the 2D explorer below—or
—and watch whether the ranking changes.
Lab 01 · Vector Space Intuition
2D Semantic Space & Cosine Similarity Probe
Drag the orange query arrow to see how angle θ drives ranking and which documents get dropped by score_threshold = 0.65.Illustrative: real vectors have 1,024+ dims, projected here to 2D
Threshold: 0.65
Top-K Similarity Ranking (cos θ = (A · B) / (‖A‖ ‖B‖))
Magnitude dropped 55%, yet every score and rank stayed identical
Cosine compares direction only. Long and short passages don't gain an unfair advantage from vector magnitude, which is why Cosine is the default metric for text search.
Choosing an Embedding Model: MTEB Is a Starting Point, Not the Finish Line
MTEB overall averages blend classification, clustering, and other tasks irrelevant to RAG. Start with the Retrieval sub-score, then decide based on Recall@5 on your own query set.
MTEB (GitHub repo) spans over a hundred datasets, and the Hugging Face live leaderboard is usually the first stop. The catch: sorting by the headline overall score often picks the wrong model.
Predict FirstOn the March 2026 English MTEB leaderboard, NV-Embed-v2 scores 72.31 overall while Gemini Embedding 001 scores 68.32. For an enterprise knowledge base, which should you test first?
Under the hood: NV-Embed-v2 scores only 62.65 on Retrieval, whereas Gemini Embedding 001 hits 67.71. The overall leader is actually weaker at retrieval.
Lab 02 · Selection Trade-Off Simulator
Three Selection Strategies, Three Outcomes
Pick the strategy you'd reach for first and inspect the engineering consequences.
Only the two models with cited leaderboard numbers are shown. Dimensions from official model cards.
Select a strategy above
This panel shows which model gets picked, how it performs on retrieval, whether you must rebuild the Qdrant collection, and what to do next.
Why do leaderboard winners underperform on private corpora? Three main reasons (see Pinecone's embedding rundown): MTEB tests general text like news and Wikipedia whose vocabulary diverges from internal docs; some models have training overlap with public benchmarks; and aggregate averages hide per-task variance.
Smaller models routinely beat larger ones in embedding retrieval. BAAI's BGE-M3 (arXiv:2402.03216, model card) has only 568M parameters yet supports 100+ languages and an 8,192-token context window. Alignment between training objectives and retrieval matters far more than parameter count. For multilingual and Chinese workloads, Alibaba's open-source Qwen3-Embedding family (0.6B / 4B / 8B, official blog) is also worth benchmarking.
End-to-End Ingestion and Search with Qdrant
Batch your embedding requests and store the original text in the payload during ingestion; at query time, combine limit with score_threshold and refuse cleanly when the result array is empty.
Among vector databases, Qdrant is written in Rust, launches with a single Docker command, ships with a built-in Web Dashboard (http://localhost:6333/dashboard), and provides a strongly typed TypeScript SDK (@qdrant/js-client-rest)—scaling cleanly from local prototype to production.
API Pricing & Dimensions Reference (Default Book Setup)
The examples in this chapter default to Alibaba Cloud Model Studio's text-embedding-v3 (¥0.5 / 1M tokens, 1,024 dims, 8,192 context; billing docs). Other common options: OpenAI text-embedding-3-small ($0.02 / 1M, 1,536 dims; docs), text-embedding-3-large ($0.13 / 1M, 3,072 dims), Voyage AI voyage-4-lite ($0.02 / 1M, 1,024 dims; pricing), and Google Gemini Embedding 001 ($0.15 / 1M, 3,072 dims; Vertex AI pricing).
Two engineering details trip up almost every first ingestion script: first, pack multiple texts into a single input array (batch upsert) instead of one HTTP call per chunk, or you will hit rate limits fast; second, store the raw text alongside source and section inside each point's payload, otherwise a vector hit gives you an ID with no text to hand back to the LLM.
Full TypeScript implementation: Qdrant batch upsert & filtered search (embed-and-store.ts)
import { QdrantClient } from "@qdrant/js-client-rest";
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: process.env.ANTHROPIC_API_KEY, // Reuses DashScope API key
baseURL: process.env.EMBEDDING_BASE_URL,
});
const qdrant = new QdrantClient({ host: "localhost", port: 6333 });
const COLLECTION = "knowledge_base";
const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL ?? "text-embedding-v3";
const VECTOR_SIZE = 1024; // Must match the embedding model output dimension
const BATCH_SIZE = 50;
async function ensureCollection() {
const { collections } = await qdrant.getCollections();
if (!collections.some((c) => c.name === COLLECTION)) {
await qdrant.createCollection(COLLECTION, {
vectors: { size: VECTOR_SIZE, distance: "Cosine" },
});
}
}
async function ingestChunks(chunks: Array<{ text: string; metadata: { source: string; section?: string; chunkIndex: number } }>) {
for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
const batch = chunks.slice(i, i + BATCH_SIZE);
const response = await openai.embeddings.create({
model: EMBEDDING_MODEL,
input: batch.map((c) => c.text), // Batch requests to reduce RPM pressure
});
const points = batch.map((chunk, j) => ({
id: i + j,
vector: response.data[j].embedding,
payload: {
text: chunk.text,
source: chunk.metadata.source,
section: chunk.metadata.section ?? "",
chunkIndex: chunk.metadata.chunkIndex,
},
}));
await qdrant.upsert(COLLECTION, { points });
}
}
async function search(query: string, limit = 5, scoreThreshold = 0.65, sourceFilter?: string) {
const res = await openai.embeddings.create({ model: EMBEDDING_MODEL, input: query });
const queryVector = res.data[0].embedding;
const results = await qdrant.search(COLLECTION, {
vector: queryVector,
limit,
score_threshold: scoreThreshold, // Drop anything below the quality floor
with_payload: true,
...(sourceFilter ? { filter: { must: [{ key: "source", match: { value: sourceFilter } }] } } : {}),
});
if (results.length === 0) {
return "No relevant context found to answer this question."; // Refuse instead of feeding noise to the LLM
}
return results;
}
Top-K, Similarity Scores, and Chunk Quality
In production RAG, cosine scores of 0.70–0.80 are already strong. When scores stay stubbornly low, inspect whether your chunks mix too many topics before swapping embedding models.
Under Cosine distance, higher Qdrant score values mean higher similarity (Qdrant Search docs). Because conversational user queries rarely match formal doc phrasing word for word, scores above 0.85 are uncommon.
If precise queries fail to break 0.70, the bottleneck is usually bloated, mixed-topic chunks rather than a weak model. Cram a dozen concepts into one chunk and its embedding collapses into a diluted "average meaning," pulling down similarity for every single question.
Predict FirstIn a controlled run on the same vector-DB notes: packing 4 topics into one large chunk yields a score of 0.53 for "What is the range of cosine similarity?" Split into single-concept chunks using the exact same model—what does the score jump to?
Actual experiment output (chunk-quality-experiment.ts): it jumps from 0.53 straight to 0.85. The focused chunk contains only the cosine similarity definition, so the vector represents that concept cleanly instead of being diluted by three unrelated topics.
In Lab 03 below, switch across all 3 real benchmark queries to compare "large & mixed" vs. "small & focused" chunks, then tune limit, score_threshold, and source filtering together. Try
to see how the threshold gate triggers safe refusal.
Lab 03 · Chunk Dilution & Parameter Tuner
Chunk Semantic Dilution & Top-K / Threshold Gate
Top: Measured · chunk-quality-experiment.ts real scores. Bottom: Simulated · Qdrant filter logic synced live to the TypeScript snippet above.
Qdrant Retrieval Parameter Tuner (syncs live to the TS code above)
Result cap limit5
Quality floor score_threshold0.65
Selection & Engineering Decision Matrix
Every rule from this chapter distilled into a filterable cheat sheet. Click a scenario tag to highlight the relevant playbooks.
Default Start
Ship end-to-end with text-embedding-v3 first
Reuse your existing DashScope key with balanced 1,024-dim bilingual vectors. Getting ingestion, payloads, and search working beats agonizing over leaderboards upfront.
Multilingual & Local
BGE-M3 or Qwen3-Embedding
568M BGE-M3 also outputs 1,024 dims, so swapping from text-embedding-v3 requires no Qdrant collection rebuild. Benchmark Qwen3-Embedding-4B when you need higher accuracy.
Benchmark Trap
Retrieval split + private query eval
Ignore MTEB overall averages. Collect 100–200 real user questions and measure Recall@5 and MRR—your private corpus score is the only verdict that counts.
Low-Score Triage
Scores < 0.70? Audit chunks first
Oversized, mixed-topic chunks dilute vector semantics. Splitting a multi-topic block into single-concept chunks jumped the exact same model from 0.53 to 0.85.
Hallucination Guard
Always pair limit with score_threshold
Without a reranker, start at limit=5 and threshold=0.65. When Qdrant returns an empty array, refuse directly instead of feeding 0.5-score noise to the LLM.
Known Boundary
Dense vectors miss exact identifiers
Asking for "GDPR Article 17" when the doc says "right to erasure" (or querying error codes and SKUs) can fall short on pure dense search. That's the gap Hybrid Search & Reranking closes next.
With offline ingestion and online vector search in place, the retrieval backbone is standing. Pure dense retrieval still has one structural blind spot: exact terms, error codes, and clause numbers. The next chapter adds BM25 keyword search, Reciprocal Rank Fusion (RRF), and Cross-Encoder reranking to push both recall and precision another level higher.