RAG Chunk Overlap: 25% Recovered Split Evidence; 75% Tripled the Index

The decision is measure, minimize, or reject the fixed chunker

Measure overlap when answer-bearing evidence is being split across chunk boundaries. Keep the smallest tested value that clears retrieval and downstream answer gates. Reject a fixed-overlap design when document structure, tables, code, permissions or update behavior makes word-count windows the wrong unit.

Do not copy 10%, 20% or 50% from a framework example and call the RAG system tuned. Overlap changes index volume, duplicate candidates, context composition and update work. It is an evaluated system parameter, not harmless preprocessing.

The sharp question: how much duplication buys the missing evidence?

The common reason for overlap is sound: a statement may begin near the end of one chunk and finish in the next. With no shared span, neither unit carries the complete evidence. The unresolved engineering question is whether each extra duplicate word improves retrieval enough to justify its costs.

We isolated that boundary condition instead of asking an LLM to answer broad factual questions. The target had cause_i at word 77, bridge_i at 79 and answer_i at 80. An 80-word zero-overlap window ends immediately before the answer token.

The chronology runs from RAG architecture to local retrieval contracts

The original RAG paper appeared in 2020 and evaluated a particular architecture that combined a sequence-to-sequence model with retrieved Wikipedia passages on named knowledge-intensive NLP tasks. It established that retrieval can be an integral part of generation in that scope. It did not establish a universal enterprise chunk size or overlap percentage.

By our August 2026 review, OpenAI’s implementation guidance framed retrieval as context optimization and recommended an evaluate–hypothesize–change–evaluate loop. On September 5, 2026, we applied that loop to one deliberately narrow hypothesis: overlap should repair a known boundary split, and any larger setting should have to earn its added index volume.

The corpus makes the boundary visible

The harness generates 12 target documents and 12 paired distractors. Every document has 240 words. Targets contain a topic token, a cause token and the answer token; distractors share topic and cause language but replace the answer with decoy language. This makes a superficially relevant competitor available for every query.

Synthetic tokens are a strength for this contract test because ground truth and positions are exact. They are also its largest limitation: natural prose, headings, tokenization, embeddings and semantic ambiguity are absent.

The retriever is intentionally too simple to hide the mechanism

For each query, the script scores every chunk by unweighted frequency of three query terms: topic_i, cause_i and answer_i. It then sorts ties lexically by chunk ID and returns four chunks. There is no embedding model, approximate-nearest-neighbor index, reranker or generator.

That simplicity prevents an opaque model from becoming the explanation. The experiment tests chunk construction and candidate composition. It does not estimate production semantic retrieval quality.

Zero overlap found the answer but lost the complete evidence

At zero overlap, answer hit rate was 12 of 12, or 100%, but complete-evidence rate was 0 of 12. The retriever could return a chunk containing answer_i, yet no chunk contained both cause_i and answer_i because the fixed boundary separated them.

This distinction matters for AI systems. “The answer token appeared somewhere in top-k” is weaker than “the retrieved unit contains the evidence needed to support the answer.” Retrieval metrics must encode the evidence contract the generator actually needs.

Twenty-five percent overlap repaired every planted split

With 25% overlap, the step fell from 80 to 60 words. A chunk beginning at word 60 then covered both positions 77 and 80. Complete-evidence retrieval rose to 12 of 12 while answer hit rate remained 12 of 12.

The local finding is causal within the generated corpus: the shifted window spans the planted boundary. It does not show that 25% improves unrelated queries, natural-language retrieval or generated answers.

Fifty and seventy-five percent bought no additional retrieval win

Complete-evidence and answer hit rates remained 100% at 50% and 75% overlap. Mean unique documents in top four stayed at 2.0, mean duplicate slots stayed at 2.0, and four fixed 80-word chunks kept mean retrieved volume at 320 words.

On every metric this harness records after the 25% repair, larger overlap was flat. That is not evidence that high overlap can never help; it is evidence that it failed to earn its extra indexing work on this boundary contract.

The index bill kept rising after the quality metric stopped

Across 24 documents, zero overlap produced 72 chunks, 25% produced 96, 50% produced 120 and 75% produced 216. The highest setting therefore tripled chunk count versus zero overlap and produced 2.25 times as many chunks as the first passing setting.

Chunk count is not a complete cost model, but it is a measurable driver of embedding work, stored vectors, metadata rows, ingestion updates and candidates that an index may inspect. Price those components with the actual provider and index; do not convert our counts into a universal currency estimate.

Fixed top-k can hide duplicate-context waste

Because top-k stayed at four and each selected chunk held 80 words, retrieved words stayed at 320 across settings. That does not mean prompt quality stayed constant. Two of four slots were duplicate-document slots on average in every condition, so nominal context volume overstated source diversity.

Track distinct source, document and section identifiers beside token count. A full context window can still carry little independent evidence when neighboring windows repeat the same passage.

Do not optimize retrieval without testing generation

Our test deliberately stops before an LLM. A generator may synthesize split chunks correctly, ignore one half, over-weight duplicated wording or answer from parametric memory. None of those behaviors is measured here.

A production evaluation should pair retrieval measures with citation support, answer correctness, abstention when evidence is absent, critical-error rate, latency and cost per accepted outcome. Preserve retrieved chunk IDs so a generation failure can be traced back to candidate construction.

Choose test cases from real evidence shapes

Sample prose clauses that cross pages, heading-to-paragraph relationships, table headers and rows, code signatures and bodies, policy exceptions, footnotes, multi-column PDFs and OCR discontinuities. Include short documents, long repetitive manuals and frequently updated records.

Label the minimum evidence span needed for each answer. If the ground truth requires two distinct sections, do not mark a single overlapping chunk as the only success; measure multi-hop retrieval explicitly.

Compare structural chunking before adding duplication

A sentence, paragraph, heading hierarchy, table row or code symbol may preserve meaning with less duplication than blind word windows. Test those boundaries against the same frozen query set. Keep document and permission metadata attached through every transformation.

Fixed overlap is a fallback for uncertain boundaries, not a substitute for parsing. Reject any parser that merges content across tenant, access-control, retention or legal boundaries merely to improve semantic continuity.

Reranking and diversity are separate levers

Overlap can place complete evidence into one candidate, while a reranker can reorder candidates and a diversity rule can prevent neighboring chunks from consuming all slots. Changing all three simultaneously makes the source of improvement unknowable.

Run an ablation: baseline chunker, overlap only, reranker only, diversity only and justified combinations. Hold top-k and evaluation cases fixed, and record candidate-level scores before and after reranking.

Updates turn overlap into an operational consistency problem

A small source edit can invalidate several overlapping chunks. Define whether ingestion replaces a complete document version atomically or permits old and new neighbors to coexist. Bind chunks to source version, parser version and chunker configuration.

During rollout, query only one coherent version. Monitor orphaned vectors, duplicate versions and stale citations. A retrieval gain is not acceptable if users receive evidence assembled from mutually inconsistent source revisions.

Security and authorization must survive chunking

The model should never receive a chunk merely because its embedding is close. Apply deterministic tenant and object authorization before retrieval results enter context, and carry source identity and classification through overlap and reranking.

Do not create an overlapping window across records with different permissions. Retrieval relevance cannot compensate for cross-tenant disclosure or a prohibited source. Treat those as non-compensable deployment failures.

Predeclare the overlap gate

For each candidate setting, record evidence containment, answer-support recall, distinct sources at k, duplicate slots, index chunks, embedding work, update amplification, retrieval latency, prompt tokens, answer correctness and critical failures. Choose thresholds before looking at the final comparison.

Select the smallest setting that clears all mandatory quality and security gates. A larger overlap must demonstrate a meaningful accepted-outcome gain that repays its added operating burden; otherwise keep the smaller passing configuration.

Rollback conditions belong in the initial decision

Rollback when index size, ingestion time, stale duplicates, retrieval latency, prompt duplication or cost exceeds its ceiling without the promised answer gain. Also rollback on authorization metadata loss, mixed source versions, citation mismatch or a critical answer failure.

Keep the previous chunker version and index available until the new configuration passes a production-shaped canary. Re-indexing is a data migration; make cutover and reversal explicit rather than silently replacing the only corpus.

What would change this conclusion

A representative corpus could show that 50% or 75% overlap clears evidence spans that 25% misses. A structural parser could beat every fixed window. An embedding retriever or reranker could change duplicate composition. A downstream model could benefit from—or be harmed by—repeated context.

Those outcomes would change the local configuration decision, not the observed harness result. For this exact corpus and retriever, 25% was the first passing overlap and 75% tripled the zero-overlap index without improving any recorded top-k outcome.

Disclosures

AccessAllGPT ran this experiment locally and received no vendor API credits, private data, advance access or compensation. No external organization sponsored, reviewed or endorsed the harness or article. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with OpenAI, Meta, University College London, New York University or organizations cited. Publication-wide relationships are listed on the disclosures page.