You're building a RAG pipeline on a corpus of PDF documents. A user asks a question that requires data from a table inside one of those PDFs — but the response is hallucinated and wrong. What could be the reason behind this and how to overcome this?
Most candidates jump straight saying "the LLM started hallucinating, optimize the prompting". That's the wrong answer or rather, it's the symptom, not the cause. Answering, this type of question will separate engineers who've read about RAG from engineers who've just shipped it.
Lets look closely at the reason for the hallucination and how to overcome this issue.
The Five-Stage Failure: Where the Pipeline Actually Breaks
A RAG pipeline for PDF documents has five stages before an answer reaches the user. For plain text, all five stages work reasonably well. For tables, every single stage introduces a distinct failure mode and they compound.
Reads the document as a stream of characters, left-to-right, top-to-bottom. No awareness of table borders, column alignment, or row boundaries.
Divides flattened text at fixed intervals with no awareness of table boundaries. A 10-row table can be split into 4 different chunks, spreading headers and values across chunks that will never be retrieved together.
Encodes a flat string into a dense vector. The vector captures semantic meaning of the text — but "Q2 4.2M 18% Net Revenue" has almost no semantic signal. The embedding is too weak to reliably surface on a meaningful query.
Returns the top-k semantically similar chunks. For a query like "What was Q2 revenue?", the retriever often returns a paragraph about Q2 revenue — which scores higher than the raw data row — instead of the actual cell value.
Generates an answer from the corrupted context it was given. The LLM is not inventing from nothing — it is reacting to damaged evidence. It produces a fluent, confident, wrong answer with no uncertainty signal.
"The most dangerous RAG failure isn't when the model says 'I don't know.' It's when it returns 4.2M when the answer is 42M — off by 10x, with full confidence. You only catch it when acutal business person calls out about the board deck."
Why does cosine similarity specifically fail for table content inside PDFs? Isn't semantic retrieval supposed to find contextually relevant content?
Cosine similarity measures the angle between two vectors in embedding space — which approximates semantic relatedness. This works brilliantly for prose, because paragraphs have rich contextual language that survives flattening. Tables don't.
A table cell's meaning is positional, not semantic. The string "4.2" in isolation carries zero semantic content. Its meaning only exists in relation to its column header ("Revenue"), its row identifier ("Q2"), and its unit context ("$M"). Standard text parsers collapse all of that relational structure into a token stream before embedding even begins.
The resulting embedding vector for a flattened table row is so semantically dilute that a query for "Q2 revenue" will often retrieve a paragraph about Q2 revenue projections — which has rich language and ranks higher — rather than the sparse data row containing the actual number.
The Fix: Rebuilding the Pipeline from the Ground Up
The fix has five components, each addressing a specific failure mode. You don't need all five immediately — but skipping Stage 1 makes every other fix irrelevant.
Fix 1 — Replace your flat parser with a structure-aware extractor
No single parser works universally on real-world PDFs. The choice depends on how the PDF was created:
| Tool | Best for | Scanned PDFs | Accuracy |
|---|---|---|---|
| Camelot (lattice) | Bordered tables in financial reports, invoices | No | Excellent |
| pdfplumber | Government docs, research papers, Word exports | No | Good |
| LlamaParse | Complex mixed-layout, tables + text + charts | Yes | Excellent |
| Docling (IBM) | Self-hosted batch pipelines, enterprise | Yes | Excellent |
| AWS Textract | Scanned documents, forms, multi-column layouts | Yes | Excellent |
| PyMuPDF | Clean digital text extraction (non-table) | No | Text only |
Camelot does not work on scanned PDFs — it needs selectable text. If your document is image-based, run pytesseract or Azure Document Intelligence or AWS Textract for OCR first, then pass the output through Camelot or pdfplumber. Skipping this step gives you clean-looking garbage.
Fix 2 — Serialize tables as self-contained meaningful chunks
Once extracted, a table must be serialized into a labeled text block that carries its own context. The column headers, source document, page number, and table label all travel with the data. So, no chunk ever arrives at the LLM missing the information it needs to interpret a value.
import camelot
import pdfplumber
from pathlib import Path
def extract_tables_from_pdf(pdf_path: str) -> list[dict]:
"""
Extract tables from a PDF using Camelot (lattice mode).
Falls back to pdfplumber for borderless tables.
Returns a list of table dicts with metadata.
"""
tables = []
path = Path(pdf_path)
# Try Camelot lattice first (best for bordered tables)
try:
raw = camelot.read_pdf(
str(path),
flavor="lattice",
pages="all"
)
for i, tbl in enumerate(raw):
if tbl.accuracy > 80: # Skip low-confidence extractions
tables.append({
"dataframe": tbl.df,
"page": tbl.page,
"table_index": i,
"accuracy": tbl.accuracy,
"source": path.name,
"extractor": "camelot-lattice"
})
except Exception:
# Fallback to pdfplumber for borderless tables
with pdfplumber.open(str(path)) as pdf:
for page_num, page in enumerate(pdf.pages, 1):
for i, tbl in enumerate(page.extract_tables() or []):
import pandas as pd
df = pd.DataFrame(tbl[1:], columns=tbl[0])
tables.append({
"dataframe": df,
"page": page_num,
"table_index": i,
"source": path.name,
"extractor": "pdfplumber"
})
return tables
def serialize_table(table: dict) -> str:
"""
Serialize a table into a self-contained labeled text block.
Column headers and source metadata are preserved in every chunk.
This is the single most important fix in the entire pipeline.
"""
df = table["dataframe"]
header = " | ".join(str(c) for c in df.columns.tolist())
rows = "\n".join(
" | ".join(str(v) for v in row)
for row in df.values
)
return (
f"[TABLE] Source: {table['source']} | "
f"Page: {table['page']} | "
f"Table #{table['table_index'] + 1}\n"
f"Columns: {header}\n"
f"{rows}\n"
f"[END TABLE]"
)
Fix 3 — Two-path ingestion: route text and tables separately
Don't treat your PDF as one homogenous document. Build two parallel ingestion paths: one for prose text, one for tables. Tag every chunk with content_type metadata so the retriever can route intelligently at query time.
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyMuPDFLoader
from langchain.schema import Document
def two_path_ingestion(pdf_path: str) -> list[Document]:
"""
Ingest a PDF document via two separate content-aware paths.
- Text path: standard chunking with section metadata
- Table path: atomic serialized chunks, never split mid-table
"""
documents = []
# ── PATH A: Text content via PyMuPDF ──────────────────────
loader = PyMuPDFLoader(pdf_path)
raw_pages = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=80,
separators=["\n\n", "\n", ". ", " "]
)
text_chunks = splitter.split_documents(raw_pages)
for chunk in text_chunks:
chunk.metadata["content_type"] = "text"
documents.append(chunk)
# ── PATH B: Table content via Camelot / pdfplumber ────────
raw_tables = extract_tables_from_pdf(pdf_path)
for table in raw_tables:
serialized = serialize_table(table)
doc = Document(
page_content=serialized,
metadata={
"content_type": "table", # critical tag for routing
"source": table["source"],
"page": table["page"],
"extractor": table["extractor"],
"table_index": table["table_index"]
}
)
documents.append(doc)
return documents
Even with clean, serialized table chunks, pure vector search has a critical weakness: it optimizes for semantic similarity, not keyword precision. When a user asks "What was Q2 2024 net revenue?", they're using exact terms — "Q2", "2024", "net revenue" — that need to match exactly in the chunk.
An embedding model can map "net revenue" and "turnover" to similar vectors and miss the exact row that uses different terminology. BM25 keyword search catches these exact matches that vector search drops.
Fix 4 — Hybrid BM25 + vector Search retrieval techinique with Reciprocal Rank Fusion
Combine sparse keyword retrieval (BM25) with dense vector retrieval and merge results using Reciprocal Rank Fusion (RRF). BM25 catches exact term matches; vector search catches semantic intent. Together they produce reliable recall for both types of query.
from langchain.retrievers import BM25Retriever, EnsembleRetriever
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
def build_hybrid_retriever(documents: list, k: int = 6):
"""
Build a hybrid BM25 + vector retriever.
BM25 handles exact keyword match for table terms.
Vector handles semantic / paraphrased queries.
RRF merges both rankings.
"""
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# Vector retriever
vectorstore = Chroma.from_documents(documents, embeddings)
vector_retriever = vectorstore.as_retriever(
search_kwargs={"k": k}
)
# BM25 keyword retriever
bm25_retriever = BM25Retriever.from_documents(documents)
bm25_retriever.k = k
# Ensemble with equal weights — tune based on your query mix
hybrid_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, vector_retriever],
weights=[0.5, 0.5]
)
return hybrid_retriever
Fix 5 — Constrain the LLM with a strict grounding prompt
Even with clean retrieval, you need to close the final gap at generation. A standard prompt lets the LLM fill gaps with its parametric memory. For numerical table queries, that's fatal. Use a system prompt that makes interpolation impossible.
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
TABLE_SYSTEM_PROMPT = """You are a precise data retrieval assistant.
You answer questions strictly from the table data provided in the context.
Rules you must never break:
1. Only cite numbers explicitly present in the table context.
2. If the exact value is not in the context, respond: "Not found in the provided documents."
3. Never interpolate, estimate, average, or infer numerical values.
4. Always cite the source file, page number, and column you retrieved the answer from.
5. If multiple tables contain relevant data, cite each separately.
"""
def build_table_rag_chain(retriever):
llm = ChatOpenAI(
model="gpt-4o",
temperature=0 # Non-negotiable for table queries
)
prompt = ChatPromptTemplate.from_messages([
("system", TABLE_SYSTEM_PROMPT),
("user", "Context:\n{context}\n\nQuestion: {question}")
])
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
chain = (
{
"context": retriever | (lambda docs: "\n\n".join(d.page_content for d in docs)),
"question": RunnablePassthrough()
}
| prompt
| llm
| StrOutputParser()
)
return chain
After implementing all five fixes — structure-aware parser, serialized chunks, two-path ingestion, hybrid retrieval, and constrained generation, table query accuracy on financial PDF corpora improves from effectively random to reliably grounded. The "4.2M vs 42M" class of error disappears because it was never an LLM problem to begin with.
Bonus Tips:
Why can't we gohead and "just extract the tables and use Text-to-SQL."
Text-to-SQL is a legitimate approach when you have a clean, consistent schema - like a relational database with typed columns. But PDF tables are the opposite of that. The schema itself must be inferred from the document, and that inference is unreliable.
Here's what breaks: a 50-page annual report might have 30 tables with different column names for the same concept across pages ("Revenue", "Net Sales", "Total Revenue", "Turnover"). Text-to-SQL has no way to resolve that unless you build a schema normalization layer — which is itself a hard NLP problem.
There's also a governance risk: a Text-to-SQL system allows users to query table structures they may not fully understand, which can expose metadata or column names that contain sensitive information. For PDF-native workflows, structured RAG retrieval with serialized chunks is the more robust path. Text-to-SQL belongs in the data warehouse layer, not the document layer.
"The hallucination isn't coming from the LLM — it's coming from the parser. By the time the model sees the data, the table structure has already been destroyed. Fix the ingestion layer first."
- Use Camelot lattice or Docling along with Azure Doc intelligence if needed
- Serialize every table as an atomic labeled chunk before embedding
- Treat text and tables through separate ingestion paths, maintain metadata of source
- Add BM25 Keyword + Sementic = hybrid retrieval for exact numerical term matching
- Set LLM temperature=0 and make source grounding in your system prompt
The Fixed Pipeline at a Glance
Structure-aware extraction preserves row-column relationships. Each table extracted as a DataFrame with headers intact.
Tables serialized as single labeled blocks with headers. Text splits at sentence boundaries. Content-type metadata tagged on every chunk.
Serialized table chunks now carry structural context — "Source: Q2_Report.pdf | Page: 4 | Columns: Quarter | Revenue | Net Margin" — giving the embedding meaningful signal to work with.
Hybrid retrieval combines exact keyword matching (BM25) with semantic similarity. Numerical values, dates, and column names are no longer missed by vector-only search.
Strict grounding prompt prohibits interpolation or estimation. LLM cites source file, page, and column for every numerical answer. Returns "Not found" rather than guessing.
Frequently Asked Questions
These are the follow-up questions that come up most often — in interviews, in code reviews, and in production postmortems.
Yes. LlamaParse is a GenAI-native parser that handles scanned PDFs via built-in OCR, as well as complex layouts with mixed text, tables, and images. It is a managed API (not self-hosted). For self-hosted alternatives with similar capability, Docling from IBM Research is the strongest open-source option, using the DocLayNet layout model and TableFormer for table structure recognition.
Two-path ingestion is an architecture pattern that routes PDF content by type during the ingestion stage. Text blocks follow a standard chunking and embedding path. Tables are extracted structurally, serialized as atomic labeled chunks with column headers intact, and indexed separately with a content_type: "table" metadata tag. At query time, a router determines whether to search the text index, the table index, or both.
Temperature controls the randomness of LLM output. At higher temperatures, the model samples more freely from its probability distributionwhich means it can interpolate or estimate values not present in the retrieved context. At temperature=0, the model always picks the highest-probability token, making it maximally faithful to the literal context. For numerical answers from tables, this eliminates the interpolation class of hallucination.
Reciprocal Rank Fusion (RRF) is a rank aggregation algorithm that combines the result rankings from multiple retrieval systems. For each document, RRF computes a score based on its position in each ranked list, then sums them. This lets you merge BM25 keyword rankings (which catch exact numerical matches) and vector similarity rankings (which catch semantic intent) without needing to tune raw score scales, which differ between retrieval methods.
Conclusion
Every RAG failure on a PDF table is an ingestion problem wearing an LLM costume. Teams spend days tweaking prompts and swapping models when the fix was always one level upstream — in how the document was parsed.
The pattern I've shipped in production is always the same: structure-aware extractor → atomic serialized chunks → two-path indexing → hybrid retrieval → grounded generation prompt. It takes a day to implement correctly. It saves weeks of debugging wrong answers.
If you take one thing from this post: before you blame the LLM, print the chunk that was actually retrieved and look at it. If it's broken, the answer was always going to be wrong — and no amount of prompt engineering fixes a corrupted context.
0 Comments