Chunking Strategy for RAG: The Decision That Affects Everything Downstream

Most RAG tutorials introduce chunking as a solved problem: split your documents into 512-token chunks with 50-token overlap, embed them, and you're done. This works adequately for simple homogeneous corpora — a collection of similar-length documents with uniform structure. It fails in significant ways on the heterogeneous enterprise data that Agentica is designed to work with.

Why Fixed-Size Chunking Fails

The fundamental problem with fixed-size chunking is that it's document-structure-agnostic. A contract clause, a financial table row, and a paragraph of narrative prose have very different natural boundaries and very different semantic density. Splitting all of them into equal-sized chunks means contract clauses get split mid-sentence, table rows lose their header context, and narrative paragraphs get artificially truncated.

The downstream effects are severe: a chunk that contains the end of one clause and the beginning of another looks like it discusses two different topics and retrieves poorly for either. A table row chunk without header context is uninterpretable without retrieval of the header chunk, but the header chunk may not rank highly enough to be included in the top-k results. Truncated narrative chunks produce answers that look right but miss the qualification or exception that appeared in the next sentence.

Structure-Aware Chunking

Agentica's chunking pipeline starts with document structure detection. PDFs, Word documents, and HTML have markup or layout information that reveals natural structure: headers, sections, tables, lists, footnotes, captions. The chunking strategy adapts to the detected structure rather than ignoring it.

For structured documents: sections become the primary chunk unit, with headers attached to their content. Tables are kept intact or split along row boundaries with headers preserved. Lists are kept together when small, split at item boundaries when large. For unstructured documents: paragraph boundaries are respected where possible, with chunks extended or shortened to align with sentence boundaries rather than token counts.

Semantic Chunking

Structure-aware chunking handles documents with explicit structure well but still produces poor results on dense, uniformly formatted documents — think legal agreements, technical specifications, or research papers where the semantic density is high and uniform throughout. For these, semantic chunking produces better results.

Semantic chunking embeds rolling windows of the document and identifies points where semantic similarity drops significantly between adjacent windows — these drops correspond to topic transitions, which are the natural chunk boundaries. The resulting chunks have variable size but consistent topical coherence. They retrieve better because the chunk's embedding accurately represents a single coherent topic rather than a blend of two adjacent topics.

Hierarchical Chunking

The most sophisticated chunking approach for enterprise documents is hierarchical: create chunks at multiple granularity levels — document, section, paragraph — and index all levels. At retrieval time, start with coarse chunks to identify relevant sections, then drill down to fine chunks for precise context extraction. This is expensive in storage (the same content is indexed multiple times) but produces significantly better recall on complex queries that require both broad context (what section is this in?) and specific content (what does the specific clause say?).

Agentica uses hierarchical chunking for document types where it's known to matter — contracts, policies, technical documentation — and simpler strategies for document types where it doesn't add value. The chunking strategy is configurable per document type in the RAG server configuration, allowing organizations to tune based on their specific corpus characteristics.