What a Sentence Forgets: Preserving Context in Document Ingestion

Retrieval systems depend on a step that receives comparatively little attention. Before a document can be searched, quoted, or used to answer a question, it must be divided into pieces small enough to index and small enough to pass to a model.

The division itself is trivial to implement. Preserving meaning across it is not.

A sentence removed from page 40 of a document no longer knows that it came from page 40. It does not know which section head it sat under, what its pronouns referred to, or what the statute it cites actually says. The document held all that information. The extracted sentence does not hold any of it, and nothing downstream can recover it.

In this blog post, I document five faults observed in a production document-ingest system — together with the measurements taken from it — present a summary of the relevant research, and provide five remediations of those faults. 

Consider a single extracted claim:

“It holds up to 128 keys.”

The referent for it appeared in a heading three lines above the sentence, and the heading was not included in the extracted piece. The claim is now unusable, though it remains grammatically well formed and will be returned by a search as though it were complete.

This is the general case. The four faults that follow are all variations of it.

This fault is addressed by remediations 1 and 2.

Dividing a PDF page by page is an intuitive approach, because a page is a meaningful unit to a human reader. It is not a unit of meaning to a parser.

Measured across the 590 pages of a 686-page document that yielded any text at all, 73% of pages began in the middle of a sentence and 78% ended in the middle of one. In one instance, a page boundary divided the word standards, and two separate model invocations were each subsequently asked to derive a fact from one half of the word.

The consequence persists well beyond the ingestion run. When that document was processed using page-based division, 43% of the resulting stored claims did not end in a full stop. These are sentence fragments that were recorded as facts, and they are returned in answer to questions:

“total income at the rate as specified in any Central Act for relevant tax year.”

The same document, reprocessed four days after the division logic was corrected, produced a figure of 1%. It is worth noting that sentence-based division produced fewer pieces than page-based division, not more.

This fault is addressed by remediation 3.

Nearly every parser already locates headings, since heading detection is how it determines that one block has ended and another has begun. Having located them, most implementations then discard the heading text.

The structural information therefore passes through the parser and is deleted a line later. 

Retaining it requires no additional reading of the document and no additional inference, because the information is already available at the point where it is dropped. In one production codebase, the entire remediation consisted of a stack maintained during a traversal that was already being performed.

This fault is addressed by remediations 1 and 2.

A heading does not terminate in a full stop. Consequently, text-extraction logic that segments on sentence-final punctuation will join the heading to the paragraph beneath it and store the result as a single unit. That unit then becomes the evidence quotation:

“EdgeKV Key limits A namespace holds up to 128 keys.”

No such sentence appears on the source page. Two headings and one paragraph were concatenated into a single assertion. On one measured page, 11 section headings were stored as stand-alone facts, each attributed to the document as something it had asserted.

This fault is addressed by remediation 4, and guarded at page boundaries by remediation 3.

Documents often refer to themselves and to one another, and statutory text does so more than most:

“Irrespective of anything contained in section 67, where the assets of a company are distributed …”

The sentence is accurate but cannot be used in isolation. The string “section 67” occupies the text in the same manner as any other words, so the claim presents as complete and carries no indication that it is contingent on material elsewhere.

Across 3,651 stored claims, 1,188 of them — approximately one in three — named some other part of the document or another document entirely. There were 1,665 references in total, of which 493 were distinct, comprising sections, subsections, schedules, chapters, and named statutes.

Three distinct operations are frequently conflated here, and the distinction matters. 

1. Recording the reference is free and carries no risk, because the reference is the document's own text. 

2. Resolving the reference is a retrieval operation that requires its own evidence and can fail. 

3. Substituting the reference with its content — replacing “section 67” with what section 67 states — constitutes a model that is editing a statute, and produces an artifact that nothing downstream can distinguish from the original.

This fault is addressed by remediation 5.

Decontextualization is defined as the task as rewriting a sentence such that it survives removal from its context without alteration of meaning. The most useful contribution of this research is the observation that certain sentences cannot be rewritten in this way, and that identifying them is itself part of a correct result: The authors report that roughly 40% of sentences were classified either as infeasible to decontextualize or as requiring no edit at all. 

A citing study reports a relative improvement of up to 30% in mean reciprocal rank for decontextualized evidence over verbatim sentences. Mean reciprocal rank is the average of 1 divided by the position of the first correct result across all queries, so a score of 1.0 means the right answer always came back first, 0.5 means it typically sat second, and anything below about 0.1 means users are scrolling past 10 results before finding it.

The LLM–based method Claimify extracts claims from a question-answer pair in four stages: 

1. Construction: The first stage splits the answer into sentences and constructs a context for each one, where that context is a configurable combination of surrounding sentences and document metadata, including the header hierarchy (remediation 1 arriving under another name). 

2. Selection: Selection then discards material that asserts nothing. 

3. Disambiguation: Disambiguation determines whether the sentence is ambiguous — and, when the surrounding context cannot resolve the ambiguity, declines rather than guessing. 

4. Decomposition: Decomposition produces the claims. The authors report that 99% of resulting claims are entailed by their source sentence.

DnDScore serves as a caution against applying both decomposition and contextualization without care. Decomposition renders each fact simple; contextualization renders each fact larger. Applying both leaves the evaluator unable to determine which component is being verified.

Heading-aware chunking attaches the enclosing section headings to each piece. It remains the least expensive intervention available and fails only where the source document contains no headings; working implementations include Docling's HybridChunker, LangChain's MarkdownHeaderTextSplitter, and LlamaIndex's MarkdownNodeParser. 

HiChunk fine-tunes a language model to recover multilevel document structure. Its more significant contribution is the accompanying benchmark, HiCBench, which the authors introduced on the grounds that existing test sets could not distinguish good chunking from poor chunking, because the supporting evidence was too sparse for boundary placement to affect the score.

RAPTOR clusters pieces, summarizes each cluster, then clusters the summaries recursively. Retrieval from the resulting tree, coupled with GPT-4, improved the best prior result on the QuALITY benchmark by 20 points of absolute accuracy.

Contextual Retrieval has a model that generates a short situating note for each piece and indexes the note alongside it, reporting that the top-20 retrieval failure rate fell by 49%, and by 67% in combination with re-ranking. 

Late chunking processes the entire document through a long-context embedding model and applies the chunk boundaries afterward, at the mean-pooling step, so that each piece's vector reflects the whole document without any new text being generated.

Three systems address the underlying problem directly. 

GROBID restructures PDFs into TEI XML, though with a particular focus on technical and scientific publications rather than documents in general. 

DocParser parses the complete document structure, including nested figures, tables, and table cell structures.

Docling converts PDFs, Office files, HTML, and images into a single document model that preserves reading order, tables, and provenance. Its chunker emits a section path and page-level bounding boxes for every chunk, which makes it the closest thing available to a reference implementation of the remediations that we discuss next.

Traverse the document in order, maintaining a single list of currently open headings. On encountering a heading of level N, pop all entries at level N or deeper, then push the new heading; every emitted block receives a copy of the list. Heading levels derive from h1 through h6 in HTML, or from clustered font size and weight in a PDF. No additional traversal is required, since the parser is already performing this walk.

The heading stack is a property of the document rather than of the page. An implementation that iterates over pages and instantiates a fresh parser for each will discard the stack several hundred times over a long document; the iteration should occur inside a single traversal that owns it. Running headers must be suppressed first, otherwise the title repeated at the head of each page will be pushed onto the stack as a genuine heading.

Join before splitting: Wherever the final block on a page does not terminate in a full stop and the following page does not open with a heading, concatenate them and repair any hyphenation introduced by the line break. Then apply a sentence splitter — pysbd, spaCy, and the ICU break iterator are all suitable and none is a model — and pack complete sentences up to the size limit.

The heading test is a stopping condition rather than a context mechanism. A page with a successor that opens with a heading has genuinely ended, notwithstanding the absence of punctuation, and an unguarded join would weld that heading to the fragment above it, reproducing fault 4 by a different route. This change reduced the fragment rate from 43% to 1%.

The underlying defect is flattening: Markup is stripped and the enclosed text retained, so a heading and its following paragraph become a single string. Type blocks before extracting their text, so that a heading is written to the piece's metadata rather than its body. For data already stored, a serviceable detector is a sentence containing a capital letter mid-string with no preceding full stop.

Match the citation forms the corpus actually uses — section 67, subsection (17), Schedule III, Chapter VI-A, and the names of other statutes — using a pattern list and a gazetteer. Store the matches in a field adjacent to the section path, so that each piece carries both its own text and an explicit list of its dependencies. Resolution belongs at query time, where it has access to its own evidence and can fail visibly.