feat: HierarchicalTextSplitter — header-aware chunking with verbless sentence filtering

[jexp]

Summary

Adds HierarchicalTextSplitter, a new TextSplitter based on the chunking strategy from Min et al., arXiv:2507.03226 — Towards Practical GraphRAG: Efficient KG Construction and Hybrid Retrieval at Scale.

Splits documents at section headers first, then recursively at a configurable character limit with overlap, and optionally drops verb-free sentences as a cheap pre-filter before chunking.

Related: the companion PR #547 (SpacyEntityRelationExtractor) uses this splitter in its example pipeline.

Header detection strategies

Strategy	Description	Best for
"markdown" (default)	Lines starting with #	MarkdownLoader, LiteParseLoader(output_format="markdown")
"capitalization"	Short ALL_CAPS or Title Case lines without terminal punctuation	Plain-text loaders
"blank_line"	Short lines surrounded by blank lines	Plain-text loaders
"spacy_verbless"	SpaCy-detected verbless sentences	Mixed-format documents

Constructor

HierarchicalTextSplitter(
    max_chunk_size: int = 2048,
    chunk_overlap: int = 200,
    header_strategy: str = "markdown",  # "markdown" | "capitalization" | "blank_line" | "spacy_verbless"
    model: str = "en_core_web_sm",      # only loaded for spacy_verbless or drop_verbless_sentences=True
    drop_verbless_sentences: bool = True,
)

Usage

from neo4j_graphrag.experimental.components.text_splitters import HierarchicalTextSplitter

splitter = HierarchicalTextSplitter(header_strategy="markdown", max_chunk_size=2048, chunk_overlap=200)
chunks = await splitter.run(text)

Pairs naturally with LiteParseLoader(output_format="markdown") from PR #534.

Test plan

• uv run pytest tests/unit/experimental/components/text_splitters/test_hierarchical_splitter.py -v (20 unit tests, spacy mocked — no model download)
• python -m spacy download en_core_web_sm && uv run pytest tests/unit/experimental/components/text_splitters/test_hierarchical_splitter_integration.py -v (2 integration tests)
• uv run python -c "from neo4j_graphrag.experimental.components.text_splitters import HierarchicalTextSplitter; print('ok')"

[matteomedioli]

Great work on this, @jexp! thanks for putting it together! I spent some time testing it locally and left some minor comments/nits along the way. None of them are blocking, but I'd value your take on a couple of them when you get a chance. Thanks again for the contribution!

[matteomedioli]

With header_strategy="spacy_verbless", the header this strategy just detected gets deleted by the default verb filter (drop_verbless_sentences=True):

splitter = HierarchicalTextSplitter(header_strategy="spacy_verbless")  # drop_verbless_sentences=True by default
await splitter.run("Introduction. This section covers the basics. Conclusion. This wraps up.")

Step 1: _split_at_spacy_verbless finds "Introduction." and "Conclusion." (short, no verb, followed by long sentence) → uses them as section boundaries.
Step 2: sections become chunks.
Step 3: verb filter runs on the final chunks, sees "Introduction." has no verb → deletes it. Same for "Conclusion."
Result: 2 chunks, no "Introduction", no "Conclusion".

Maybe we should skip verb filter on text just used as a section boundary, or warn when both options are combined together.

[matteomedioli]

Nit: chunk_overlap width can drift a bit.

Whitespace-snapping (so words don't get chopped) shifts where each chunk ends, but the next chunk's start is still fixed (start + (max_size - overlap)), so the shared region between two chunks isn't always exactly chunk_overlap chars — just close to it. Nothing is lost, content still overlaps either way.

text = "The cat sat on the warm mat today."
splitter = HierarchicalTextSplitter(max_chunk_size=15, chunk_overlap=5, header_strategy="markdown")

# chunk 0: 'The cat sat on '
# chunk 1: 't on the warm '
# chunk 2: 'arm mat today.'
# gap 0->1: tail='t on ' == head='t on '   -> matches
# gap 1->2: tail='warm ' != head='arm m'   -> off by one, drifted

Nit, not blocking at all, maybe might be worth a quick docstring note so it's clear the overlap is "approximate" rather than exact.

[matteomedioli]

Nit: text reformatted even when nothing is dropped

text = "# Section\nThe team shipped the release.\nThey tested it thoroughly.\n"
HierarchicalTextSplitter().run(text)  # every sentence has a verb

Step 1: verb filter runs anyway (default True, unconditional).
Step 2: splits into sentences, keeps all (all have verbs).
Step 3: rejoins with " ".join(...).
Result: output != input -> the newline between sentences becomes a single space, even though nothing was filtered.

Possible solution: return the original text unchanged when len(kept) == len(list(doc.sents))

[matteomedioli]

Nit, very edge case:

text = "This code selects records.\nSELECT SINGLE F1 FROM VBBS INTO WA\nWHERE ID = X\nDone."
HierarchicalTextSplitter(header_strategy="capitalization").run(text)

Both code lines get misread as headers here:
SELECT SINGLE F1 FROM VBBS INTO WA
WHERE ID = X
Splits the snippet line by line instead of keeping it together. low risk, just flagging.

[matteomedioli]

defined, never referenced

[matteomedioli]

Nit: loads NER/lemmatizer/etc even though only sentence boundaries + POS tags are used. We can consider spacy.load(model, disable=["ner", "lemmatizer", "attribute_ruler"]).

[matteomedioli]

Nit: calls _filter_verbless once per chunk in a Python loop, which calls nlp(text) (line 269) each time: spaCy processes one chunk at a time instead of as a batch.

Possible other approach:

# spacy batch instead of one nlp() call per chunk:
docs = list(nlp.pipe(raw_chunks))
for raw, doc in zip(raw_chunks, docs):
    kept = [s.text for s in doc.sents if any(t.pos_ in ("VERB", "AUX") for t in s)]
    filtered = " ".join(kept) if kept else raw
    if filtered.strip():
        chunks.append(TextChunk(text=filtered, index=len(chunks)))

[matteomedioli]

unused, remove for linter