Retrieval-Augmented Generation (RAG) often comes down to a deceptively simple question: how much text should you retrieve for a user query?
Retrieve very small chunks, and semantic search can become precise—but the retrieved text may lack the surrounding context needed by the LLM. Retrieve large chunks, and you preserve context—but you may also introduce irrelevant information, dilute the relevant passage, and increase the amount of text passed to the model.
LangChain’s ParentDocumentRetriever takes a different approach: search using small child chunks, but return their larger parent documents.
This makes it a useful technique for building more context-aware RAG pipelines without giving up the precision of smaller retrieval units.
In this guide, we’ll understand how ParentDocumentRetriever works, why chunk size matters in RAG, how to implement it with LangChain and Chroma, and when it makes sense compared with a conventional vector retriever.
What Is Parent Document Retrieval?
Parent document retrieval is a RAG strategy where a document is represented at two levels:
- Child chunks — small pieces optimized for semantic/vector search.
- Parent chunks — larger pieces that provide context to the LLM.
When a query arrives, the retriever:
- Embeds the query.
- Searches the vector store against the smaller child chunks.
- Identifies the parent documents associated with those child chunks.
- Retrieves the larger parent chunks from a document store.
- Returns those parent chunks to the application or LLM.
The important distinction is:
The text used for retrieval doesn’t have to be the same text returned to the LLM.
A simplified representation looks like this:
Original document
│
▼
Parent chunks
(2000 characters)
│
├───────────────┐
▼ ▼
Child chunks Child chunks
(200 characters) (200 characters)
│ │
└───────┬───────┘
▼
Vector Store
│
User Query
│
▼
Match child chunks
│
▼
Retrieve parent chunks
│
▼
LLMThis is particularly useful when a small passage contains the answer, but the surrounding text is necessary to interpret that answer correctly.
Why Chunk Size Matters in RAG
Chunking is one of the most important design decisions in a RAG system.
Suppose your source document contains this policy:
Employees traveling for business are eligible for a $75 daily meal allowance. This allowance covers breakfast, lunch, and dinner. It cannot be used for alcohol. Receipts are only required for individual purchases over $25, and all expense reports must be submitted within 30 days of returning from the trip.
Now consider the query:
Do I need a receipt for a $15 lunch?
A very small chunk might contain:
Receipts are only required for individual purchases over $25.That’s excellent for semantic retrieval. The query and the relevant rule are closely related.
But imagine the LLM receives only that fragment. It doesn’t know from the retrieved context that the policy is specifically discussing business travel expenses, or that the meal allowance is $75 per day.
On the other hand, if we retrieve an extremely large section—perhaps an entire travel-policy document—we preserve plenty of context, but potentially at the cost of introducing unrelated policies about flights, hotels, mileage, reimbursements, and approval workflows.
So there is a practical trade-off:
Smaller chunks
│
├── Better retrieval precision
├── Less irrelevant text
└── Greater risk of losing surrounding context
Larger chunks
│
├── More surrounding context
├── Better preservation of relationships
└── More irrelevant information/noiseThe problem isn’t simply that large chunks are “bad.” Rather, retrieval granularity and generation context have different requirements.
ParentDocumentRetriever addresses that distinction directly.
How LangChain ParentDocumentRetriever Works
LangChain’s ParentDocumentRetriever maintains a relationship between the smaller chunks used for retrieval and their larger parent documents.
Conceptually, the process looks like this:
1. Split the source into parent chunks
For example:
Parent chunk = 2,000 charactersThese chunks retain a meaningful amount of surrounding context.
2. Split each parent into child chunks
For example:
Child chunk = 200 charactersThese smaller fragments are better suited to vector similarity search.
3. Store the children in a vector store
The child chunks are embedded and stored in a vector database such as Chroma.
Child chunk
↓
Embedding model
↓
Vector
↓
Chroma4. Store the parents separately
The parent chunks themselves are stored in a document store.
In the example below, we’ll use LangChain’s InMemoryStore.
Parent chunk
↓
Docstore5. Search using the children
When the user asks a question, the query is embedded and compared with child vectors.
"Do I need a receipt for a $15 lunch?"
↓
Query embedding
↓
Vector search
↓
Matching child chunk6. Return the associated parents
The retriever follows the relationship between the matching child and its parent.
Matching child
↓
Parent ID
↓
Parent document
↓
LLMThe LLM therefore gets a larger contextual passage even though retrieval happened at a much finer granularity.
ParentDocumentRetriever vs. Regular Vector Retrieval
A conventional RAG pipeline often looks like this:
Document
↓
Split into chunks
↓
Embed chunks
↓
Vector store
↓
Query
↓
Retrieve matching chunks
↓
LLMThe same chunks are effectively doing two jobs:
- being searchable units
- being context supplied to the LLM
That can force you to compromise on chunk size.
With ParentDocumentRetriever, those responsibilities are separated:
Document
│
Parent splitter
│
┌────────┴────────┐
▼ ▼
Parent chunks Child splitter
│ │
│ ▼
│ Child chunks
│ │
▼ ▼
Docstore Vector store
│
▼
Query
│
▼
Child retrieval
│
▼
Parent retrieval
│
▼
LLMThis gives you more flexibility over retrieval granularity.
| Approach | Search unit | LLM context | Main trade-off |
|---|---|---|---|
| Small vector chunks | Small | Small | Can lose context |
| Large vector chunks | Large | Large | More noise/irrelevant content |
| ParentDocumentRetriever | Small child | Larger parent | More storage and retrieval complexity |
A Simple LangChain Example
Let’s use a small travel-policy document.
First, define two splitters:
parent_splitter = RecursiveCharacterTextSplitter(
chunk_size=2000
)
child_splitter = RecursiveCharacterTextSplitter(
chunk_size=200
)The parent splitter determines the larger pieces eventually returned by the retriever.
The child splitter determines the smaller pieces indexed for semantic search.
The exact chunk sizes aren’t universal rules. They are starting points that should be evaluated against your documents, embedding model, retrieval quality, and LLM context window.
Setting Up the Storage Layers
Next, we need two storage mechanisms.
The vector store contains the child chunks:
vectorstore = Chroma(
collection_name="split_parents",
embedding_function=embeddings
)The document store contains the parent chunks:
docstore = InMemoryStore()This separation is central to the architecture.
Chroma answers:
Which small piece of text is semantically relevant?
The docstore answers:
What larger piece of text contains that relevant information?
Creating the ParentDocumentRetriever
Once the two storage layers and splitters are ready, create the retriever:
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=docstore,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)The retriever now knows:
- where to search for child vectors,
- where to retrieve parent documents,
- how large the parent chunks should be,
- and how finely the parents should be divided for vector search.
Adding Documents
You can then add your source documents:
retriever.add_documents(docs)LangChain handles the parent/child splitting and storage relationships.
Conceptually, one source document becomes something like:
Parent A
├── Child A1
├── Child A2
├── Child A3
└── Child A4
Parent B
├── Child B1
├── Child B2
├── Child B3
└── Child B4The child chunks are indexed for retrieval, while the parent chunks remain available through the document store.
Querying the Retriever
Now consider:
query = "Do I need a receipt for a $15 lunch?"
retrieved_docs = retriever.invoke(query)The vector search may identify the child containing:
Receipts are only required for individual purchases over $25.But instead of returning only that tiny fragment, ParentDocumentRetriever can return its larger parent context:
Employees traveling for business are eligible for a $75 daily meal allowance.
This allowance covers breakfast, lunch, and dinner. It cannot be used for alcohol.
Receipts are only required for individual purchases over $25, and all expense
reports must be submitted within 30 days of returning from the trip.The LLM can now reason over the relevant rule in context.
Does Smaller Chunk Retrieval Actually “Lose Context”?
Yes—but it is worth being precise about what that means.
Smaller chunks don’t inherently produce worse retrieval. In fact, smaller chunks can make semantic matching more precise because each vector represents a more focused piece of information.
The issue occurs after retrieval.
If a fact depends on information contained elsewhere in the document, a very small retrieved chunk may not contain enough information for the generation step.
For example:
Chunk 1:
Employees traveling for business are eligible for a $75 daily meal allowance.
Chunk 2:
Receipts are only required for individual purchases over $25.A query about a $15 business-trip lunch might strongly match Chunk 2.
But Chunk 2 alone doesn’t contain the complete context surrounding the allowance.
This is one reason hierarchical retrieval strategies can be useful: use granular units to locate information, then restore context before generation.
Do Larger Chunks Introduce Bias?
“Larger chunks introduce bias” is a little too strong as a general statement.
A better description is that larger retrieved chunks can introduce irrelevant information and competing context, which can potentially make generation less focused.
For example, imagine retrieving a 5,000-token section containing:
- meal allowances,
- hotel reimbursement,
- airfare,
- mileage,
- corporate-card rules,
- expense-report deadlines.
Only one paragraph may actually answer the user’s question.
The additional material isn’t necessarily “bias,” but it can increase the amount of information the model must process and potentially make it harder to focus on the relevant evidence.
This is especially important when:
- documents are long,
- topics change frequently within a chunk,
- multiple policies contain similar terminology,
- context windows are constrained,
- or retrieval returns several large chunks.
So a more defensible framing is:
Small chunks can improve retrieval precision but risk losing context, while large chunks preserve context but can increase irrelevant information and dilute the retrieved evidence. Parent-document retrieval attempts to balance these competing requirements.
When Should You Use ParentDocumentRetriever?
ParentDocumentRetriever is particularly useful when local semantic matches need broader surrounding context.
Good use cases include:
Policy and compliance documents
A specific rule may only make sense alongside the policy section containing its conditions and exceptions.
Technical documentation
A retrieved parameter description may need the surrounding method, configuration example, or warning.
Contracts and legal documents
A sentence can depend on definitions or conditions established elsewhere in the same section.
Knowledge bases
A precise sentence can identify the relevant section, while the parent provides the explanation required to answer the user.
Manuals and guides
A search result for one instruction may need the surrounding procedure to be useful.
It is less compelling when your source documents are already naturally small and self-contained.
ParentDocumentRetriever vs. Other RAG Strategies
Parent retrieval isn’t the only way to solve the context problem.
Other approaches include:
- Fixed-size chunking — simple and effective for many datasets.
- Semantic chunking — attempts to create chunks around semantic boundaries.
- Metadata filtering — narrows retrieval using structured document information.
- Multi-query retrieval — generates multiple search queries to improve recall.
- Contextual compression — retrieves broader documents and then compresses them to the information relevant to the query.
- Reranking — retrieves candidates first and uses a reranker to improve relevance.
In a production RAG system, these techniques can also be combined.
For example:
User query
↓
Query rewriting
↓
Child vector retrieval
↓
Parent retrieval
↓
Reranking / compression
↓
LLMThe right architecture depends heavily on the document structure and the failure modes you’re seeing in evaluation.
Complete LangChain ParentDocumentRetriever Example
Here is the complete example in one place. It uses OpenAI embeddings, Chroma for child-vector storage, and an in-memory document store for parent chunks.
from langchain.retrievers import ParentDocumentRetriever
from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_core.stores import InMemoryStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
# 1. Initialize the embedding model
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small"
)
# 2. Define the splitters
# Larger chunks that provide context to the LLM
parent_splitter = RecursiveCharacterTextSplitter(
chunk_size=2000
)
# Smaller chunks optimized for semantic search
child_splitter = RecursiveCharacterTextSplitter(
chunk_size=200
)
# 3. Create the storage layers
# Stores vectors for the smaller child chunks
vectorstore = Chroma(
collection_name="split_parents",
embedding_function=embeddings
)
# Stores the larger parent chunks
docstore = InMemoryStore()
# 4. Create the ParentDocumentRetriever
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=docstore,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
# 5. Load source documents
docs = [
Document(
page_content=(
"Employees traveling for business are eligible for a "
"$75 daily meal allowance. This allowance covers "
"breakfast, lunch, and dinner. It cannot be used for "
"alcohol. Receipts are only required for individual "
"purchases over $25, and all expense reports must be "
"submitted within 30 days of returning from the trip."
),
metadata={
"source": "travel_policy_2026.pdf"
}
)
]
# Split the documents into parent and child chunks,
# store child chunks in the vector store,
# and store parent chunks in the docstore.
retriever.add_documents(docs)
# 6. Query the retriever
query = "Do I need a receipt for a $15 lunch?"
retrieved_docs = retriever.invoke(query)
# 7. Display the retrieved parent documents
print(
f"Retrieved {len(retrieved_docs)} parent document(s):\n"
)
for doc in retrieved_docs:
print(doc.page_content)
print("\nMetadata:", doc.metadata)The key idea isn’t simply that LangChain splits a document twice.
It’s that the two representations serve different purposes:
Child chunks → Retrieval precision
Parent chunks → Generation contextThat separation is what makes ParentDocumentRetriever a useful advanced RAG technique.
Final Takeaway
Traditional vector retrieval often forces you to choose between precision and context.
Small chunks make it easier to find the exact passage relevant to a query, but they can strip away information needed to interpret that passage. Large chunks preserve more context, but they can bring along irrelevant material and make the retrieved evidence less focused.
LangChain’s ParentDocumentRetriever provides a practical middle ground:
Search small, return large.
The child chunks make semantic retrieval more granular, while their associated parent chunks give the LLM a broader piece of the original document to work with.
It isn’t a universal replacement for conventional vector retrieval, but for structured documents where individual facts depend on their surrounding context, it can be a valuable component of an advanced RAG architecture.
Further reading
If you’re evaluating this in production, don’t choose the parent/child sizes solely from rules of thumb. Test different chunk sizes against a representative set of questions and measure retrieval recall, answer correctness, context relevance, latency, and token usage.
