AI Agent Guide Interactive
Module 3 · RAG Retrieval & Grounding 12 min read · 3 interactive labs

Document Parsing & Chunking: Why RAG Fails Before Embeddings

Most teams building their first RAG system spend all their energy tuning embedding models, vector indexes, and prompts—yet retrieval still returns noisy, half-baked answers. Trace the bug upstream and you usually find broken reading order, flattened tables, or mid-sentence cuts. Once upstream ingestion is corrupted, swapping models downstream is just polishing dirty data.

30-Second TL;DR · 3 Engineering Takeaways
  1. 01 Chunkers and embeddings cannot fix broken reading order. Multi-column scrambling, flattened tables, and scanned PDFs must be solved at the parser stage.
  2. 02 There is no universal optimal chunk size. Start with a boundary-preserving Recursive baseline (with explicit punctuation preservation), then tune on your own queries.
  3. 03 Metadata has two distinct paths: payload fields power filtering and citations without altering vectors, while title prefixes enrich isolated semantics at a token cost.

Why Chunking Still Matters in the Long-Context Era

Long context solves how much you can stuff into a prompt; RAG solves what you should put in. Upstream document processing determines what your retriever can actually see.

Context windows keep growing, and managed File Search tools are now built straight into model platforms. Even so, dumping entire product manuals, HR policies, and historical tickets into every request wastes tokens and floods attention with irrelevant text. Whenever knowledge updates continuously, access permissions vary by user, or answers require verifiable citations, the system still needs to retrieve focused evidence first.

In modern AI systems, RAG acts as the knowledge layer for Agents: the Agent decides when to search, what to query, and whether to dig deeper, while the retrieval pipeline surfaces grounded DocumentChunk records.

RAG Document Preprocessing Pipeline: 4 Stages & Upstream Failure Points Architecture diagram
Main pipeline · Raw files to grounded chunks Raw Docs PDF / Markdown 1. Parse Reading order 2. Clean Denoise & NFKC 3. Chunk Boundaries + overlap 4. Metadata DocumentChunk[] High-frequency failure modes · Unrecoverable downstream Scrambled columns Regex eats headings Mid-sentence cuts No source / ACLs

PDF Parsing Is About Layout Reconstruction, Not Just Reading Bytes

Markdown and plain text can be read directly. PDFs are designed for visual presentation—positioning glyphs at exact coordinates—not for preserving logical reading order. Ordinary PDFs force parsers to guess paragraphs, columns, and table cells from coordinates, and those guesses routinely break in three places: multi-column layouts, flattened tables, and scanned images without a text layer.

Lab 01 · Upstream Parsing & Cleaning Clinic

See how PDF extraction breaks—and how a “universal” regex creates new dirty data

Inspect multi-column and table extraction failures, then toggle individual cleaning rules in Tab 3 to watch naive soft-wrap merging swallow headings and list items.

Interactive diagnostic
Failure Mode:
Parser:
Original PDF Visual Layout Human view
Extracted Text Stream Corrupted order

            

In Node.js, pdf-parse is a pragmatic starting point for clean, text-layered PDFs: it is written in TypeScript, ships built-in types, needs no native bindings, and includes basic table detection. When you need low-level glyph coordinates and font metadata, reach for Mozilla's pdfjs-dist. For complex multi-column reports or table-heavy PDFs, evaluate a dedicated parser such as LlamaParse (verify credit consumption on its pricing page against your own sample documents first).

Quick Check for Scanned PDFs & OCR QA
Try selecting text in a PDF viewer as a fast sanity check: if you cannot highlight text, you likely need OCR (though outlined fonts or copy restrictions can look similar). Always spot-check headings, numbers, proper nouns, and tables after OCR—your vector database will faithfully retrieve OCR typos without knowing they are wrong.

Three Chunking Strategies, Separators, and Overlap

Good chunking balances two opposing forces: small enough for focused embeddings, yet large enough to answer a question without missing context.

Make chunks too large and one vector gets diluted across multiple topics. Make them too small and you index sentence fragments. Cut in the wrong spot and a key fact gets split across two chunks so neither can answer the query alone.

Predict First Semantic chunking embeds adjacent sentences to detect topic shifts. In published benchmarks, does it consistently beat Recursive or Page-level chunking?
Empirical findings: Vectara's NAACL 2025 evaluation Source: Vectara NAACL 2025 ↗ found that semantic chunking did not deliver consistent gains across document retrieval, evidence retrieval, or grounded generation, leaving its extra embedding cost unsupported. NVIDIA's 2025 chunking study Source: NVIDIA 2025 ↗ likewise showed that optimal chunk size varies by dataset, page-level chunking achieved the best overall average, and 15% overlap helped only in specific combinations.
Lab 02 · Live Chunking & Cut-Line Sandbox

Watch cut-lines move on real prose as you tune strategy, separators, size, and overlap

Runs the chapter's actual fixedSizeChunk, splitRecursive, and addOverlap algorithms in your browser. Switch strategies to spot mid-sentence cuts, dropped punctuation, and overlap overflow.

Pedagogical sandbox · char-based
Strategy:
Overlap Budget:
CHUNK_SIZE (char limit) 180 chars
OVERLAP (ratio) 15% (27 chars)
Live Cut-Line Projection (cut markers projected directly onto the source prose)
Clean boundary Mid-sentence cut Overlap borrowed tail
View TypeScript implementation synced with current parameters
chunking-strategies.ts · synced with controls

Start simple: build a Fixed-size or Recursive baseline with 256–512 tokens and 10%–15% overlap as initial candidates, then let your own evaluation queries decide. Remember that characters do not map cleanly to tokens across tokenizers—especially in multilingual or CJK text. Our runnable examples use character counts for clarity; switch to your embedding model's tokenizer in production.

For multilingual or CJK documents, explicitly include full-width sentence delimiters (。?!;,) since Chinese and Japanese lack word spaces, and always re-attach the matched delimiter after split() so adjacent sentences do not glue together.

Two Paths for Metadata Injection & End-to-End Pipeline

Never store bare chunk strings. Every chunk needs at least its source file, section path, and chunk index—and you must decide what belongs in payload vs. embedded text.

source enables document-level filtering and user-facing citations; section anchors the chunk inside a heading hierarchy; chunkIndex and totalChunks support neighbor expansion and debugging.

Two Metadata Paths: Vector Payload vs. Embedded Title Prefix Architecture diagram
DocumentChunk text: "Standard tickets: 4h SLA..." source · section · chunkIndex Path A: Vector DB Payload ACLs · metadata filters · citations Leaves embedding vector untouched Path B: Embedded Title Prefix [Doc / Section] + body text Helps isolated chunks; shifts similarity Indexed Record vector: float[] payload: { source, ... }
Lab 03 · Metadata Dual-Path X-Ray

When 2024 and 2025 policy chunks read nearly identically, how does retrieval pick the right year?

Test query: "What is the updated 2025 support ticket SLA?". Compare what gets embedded, what lands in payload, and how ranking shifts when no explicit filter is passed.

Simulated ranking
Indexed Record (Embedding Input + Payload) Prefix overhead ~ +34 chars

          
Top-2 Ranking (without explicit year filter)

Running the End-to-End Ingestion Pipeline

Chaining parsing, cleaning, overlap-reserved recursive chunking, and metadata attachment produces ready-to-embed DocumentChunk[] objects. Complete runnable scripts and test suites live in the public repository at examples/rag/document-parsing-chunking/.

Inspect the full processDocument() TypeScript pipeline
examples/rag/document-parsing-chunking/parse-and-chunk.ts
import path from "path";
import { PDFParse } from "pdf-parse";

interface DocumentChunk {
  text: string;
  metadata: {
    source: string;
    section?: string;
    chunkIndex: number;
    totalChunks: number;
  };
}

async function parsePDF(filePath: string): Promise<string> {
  const parser = new PDFParse({ url: filePath });
  const data = await parser.getText();
  return data.text;
}

async function processDocument(filePath: string): Promise<DocumentChunk[]> {
  const fileName = path.basename(filePath);
  const rawText = await parsePDF(filePath);
  const cleanedText = cleanText(rawText);

  const CHUNK_SIZE = 512;
  const OVERLAP = Math.floor(CHUNK_SIZE * 0.15);

  // Reserve overlap budget up front so final chunks never exceed CHUNK_SIZE
  const splitChunks = splitRecursive(cleanedText, CHUNK_SIZE - OVERLAP);
  const textChunks = addOverlap(splitChunks, OVERLAP);

  return textChunks.map((text, index) => ({
    text,
    metadata: {
      source: fileName,
      chunkIndex: index,
      totalChunks: textChunks.length,
    },
  }));
}
Real-World Gotcha: Compatibility Glyphs Surviving NFKC
When running the script on real PDFs, certain font-encoded compatibility glyphs can still survive normalize("NFKC"). Add source-specific character maps or upgrade the parser rather than assuming a single cleaning pass handles every PDF. Likewise, for Markdown files, parse into an AST and split by heading hierarchy first instead of piling on regexes.

Engineering Decision Matrix

Match your parser, chunker, and metadata strategy to your document structure—and verify every change with spot checks and real queries.

Parsing
Clean, text-layered single-column PDFs

Start with pdf-parse, persist both raw and cleaned text for spot-checking, and verify reading order before tuning chunks.

Parsing
Multi-column papers, dense tables, or scans

Upgrade the parser first (pdfjs-dist for custom layout logic, LlamaParse or OCR for complex layouts). Never expect chunking to repair scrambled text.

Chunking
First production baseline & tuning range

Start with Recursive (or Page-level for self-contained reports) at 256–512 tokens and 10%–15% overlap; only test Semantic chunking after measuring baseline gaps.

Cleaning & CJK
Multilingual / CJK docs & noisy extractions

Add explicit full-width delimiters (。?!;,) and preserve matched punctuation. Add cleaning rules strictly from observed sample noise.

Structure
Markdown knowledge bases & API docs

Use a Markdown AST parser to split by heading sections while keeping code fences and tables intact, then recursively split only oversized sections.

Metadata
Versioned policies & multi-tenant ACLs

Always store source, section, and chunkIndex in payload; prepend a short title prefix to embedded text only when isolated chunks lose critical context.