theindexproject

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

theindexproject

(★ 13)

No description available.

theindexproject Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# How AI Reads 1,000 Pages in Seconds — And What's Really Happening Inside

*By Abhishek Shah · abhishekshah.vercel.app · abhishek.aimarine@gmail.com*

---

You paste a 200-page PDF into an AI chat. Seconds later, it gives you a clean summary. It pulls a specific line from page 147. It even cites the section. That feels like magic.

It isn't magic. It's a pipeline. And once you understand the pipeline, the whole thing clicks — including why it still breaks in frustrating, sometimes embarrassing ways.

This article walks through exactly what happens inside an AI system when you throw a large document at it. Every step. The theory, the engineering, the failure modes, and the ideas that make it all work. By the end, you'll have a mental model solid enough to build with it, debug it, and know when to trust it.

---

## First, a Necessary Truth About How LLMs "Read"

Language models don't read the way you do.

You read a document linearly. You remember what you read on page 3 when you're on page 87. You can make inferences across chapters. You understand context that spans the whole thing.

A language model doesn't have any of that. No scroll bar. No memory across sessions. No concept of "earlier in the document" in the way humans experience it.

What it has is a **context window** — a fixed amount of text it can process in a single pass. Everything inside the context window is visible to the model, simultaneously. Everything outside it simply doesn't exist from the model's perspective.

Think of it like a spotlight. Whatever falls inside the spotlight, the model can reason about. Whatever falls outside — it's dark.

The context window is measured in **tokens**. A token isn't exactly a word. It's roughly 3–4 characters, so one word averages about 1.3 tokens. A typical English novel (80,000 words) is about 100,000 tokens. [1]

Here's where the major models stand right now:

- **GPT-4 Turbo** — 128,000 tokens (~96,000 words)
- **Claude 3 Opus** — 200,000 tokens (~150,000 words)
- **Gemini 1.5 Pro** — 1,000,000 tokens (~750,000 words)

These numbers sound massive. And they are. But bigger isn't automatically better — and this is where a critical piece of research changed how everyone thinks about long-document AI.

---

## The Problem Nobody Talks About Enough: Lost in the Middle

In 2023, researchers from Stanford published a paper called *"Lost in the Middle: How Language Models Use Long Contexts."* [2]

The finding was uncomfortable. Models perform well on information at the **beginning** and **end** of a long context. Accuracy drops significantly for content buried in the **middle** of a long document — even if the model technically "saw" it.

Why? Because of how attention works.

The transformer architecture — the foundation of all modern LLMs — computes **self-attention**: every token looks at every other token to figure out what's relevant. It's like asking every word in a document "which other words matter most to understanding you?"

Over very long sequences, the attention signal gets diluted. There's too much noise. The model has trouble maintaining the relevance signal for content that's far from the start or end of the input.

So a longer context window means more text the model can *technically fit*. It doesn't mean more text the model will actually *use well*.

This is the real problem with just dumping your entire document into a prompt. And it's why the field converged on a different solution.

---

## The Solution: RAG — Retrieval-Augmented Generation

Instead of shoving the whole document in front of the model, what if you only gave it the *relevant parts*?

That's the core idea behind **RAG — Retrieval-Augmented Generation**. [3]

The name says it all. You retrieve the relevant content first. Then you use it to augment the generation. The model sees the question, plus a focused set of relevant document chunks, and generates its answer from that.

This solves the context window problem. It solves the lost-in-the-middle problem. It makes the system auditable — you can trace exactly which parts of the document produced which answer. And it massively reduces hallucination, because the model is grounded in real source material rather than reasoning from memory.

RAG is probably the most important production pattern in applied AI right now. It underpins document Q&A systems, enterprise search, customer support bots, legal analysis tools, medical records review — anything that involves an LLM reasoning over real documents at scale.

Here's the full pipeline, step by step.

---

## Step 1: Ingestion and Parsing — Before the AI Sees Anything

Before anything intelligent can happen, the raw document has to become clean, usable text.

This sounds trivial. It's not.

A **PDF is not a text file**. It's a rendering specification — a set of instructions telling a viewer where to paint pixels on a page. There's no inherent structure like "this is paragraph 1, this is a footnote, this is a table." Extracting readable text from a PDF means reconstructing that structure from positional data, and it breaks constantly.

Common problems:
- Two-column layouts where the extractor reads across both columns horizontally instead of down each column separately
- Footnotes appearing mid-sentence because they're positionally embedded in the text area
- Headers and page numbers bleeding into the content
- Equations and formulas that are stored as images, not text
- Scanned documents with no text layer at all — just pixel images

That last one requires **OCR (Optical Character Recognition)** — software that looks at pixels and tries to recognize characters. OCR accuracy depends heavily on scan quality, font complexity, and language. Poor OCR upstream means garbled text everywhere downstream. [4]

For non-PDF documents: Word files have their own quirks (tracked changes, embedded objects, complex table structures). HTML pages have navigation elements, ads, and boilerplate mixed in with the actual content. PowerPoint decks have text scattered across slide objects in no particular reading order.

Good ingestion pipelines handle all of this. They clean, normalize, and structure the text before passing it on. Tools like **LlamaParser**, **Unstructured.io**, and **Amazon Textract** are popular production choices.

Garbage in, garbage out. This hasn't changed. Every failure at this stage propagates silently through the rest of the pipeline, showing up as confident wrong answers at the end.

---

## Step 2: Chunking — Slicing the Document Into Pieces

Clean text goes in. What comes out is a set of **chunks** — smaller, self-contained pieces of the document.

Why chunk at all? Two reasons.

First, even with large context windows, you don't want to feed 300 pages into every query. You want to feed the relevant parts. Chunking is what makes selective retrieval possible.

Second, chunking shapes what the model sees. Each chunk becomes a unit of retrieval. The quality of the chunk — whether it contains a coherent, complete idea — directly affects whether the model can reason from it.

### Fixed-size chunking

Simplest approach. Split every N tokens. Fast. Predictable. Dumb. The problem is it slices sentences and ideas in half with no regard for meaning. Chunk N ends mid-sentence. Chunk N+1 starts with the continuation. The retrieval system now has two half-ideas instead of one complete one.

### Sentence and paragraph-based chunking

Better. Split at natural semantic boundaries — sentence endings, paragraph breaks, section headers. Each chunk contains a complete thought. Much more coherent for retrieval. Tools like `spaCy` or `NLTK` can handle sentence boundary detection.

### Recursive character splitting

The approach used by LangChain's `RecursiveCharacterTextSplitter`. It tries to split on progressively smaller separators: first paragraphs (`\n\n`), then lines (`\n`), then sentences (`.`), then words. It keeps chunks within the target size while preserving as much semantic structure as possible. [5]

### Overlapping chunks

A subtle but important technique. If chunk N ends at token 512 and chunk N+1 starts at token 513, a sentence that spans that boundary gets split. No chunk contains the full thought.

Overlapping chunks solve this. Chunk N covers tokens 1–512. Chunk N+1 covers tokens 462–974 — repeating the last 50 tokens of chunk N as an "overlap margin." The complete sentence now exists in at least one chunk.

Typical overlap is 10–20% of chunk size.

**Chunk size matters a lot.** Small chunks (128–256 tokens) give high retrieval precision — you pull very targeted content, but each chunk contains limited context. Large chunks (512–1024 tokens) give more surrounding information with each retrieval, but retrieval precision is lower. Most production teams tune this experimentally based on document type and query style. There's no universal right answer. [3]

---

## Step 3: Embeddings — Turning Text Into Math

Here's where the fundamental magic lives.

Every chunk gets passed through an **embedding model**. The model converts the text into a **vector** — a list of floating point numbers, typically 768 to 3072 numbers long.

This vector is a mathematical representation of the *meaning* of the text. Not the words. The meaning.

The key property: two pieces of text that mean similar things will have vectors that point in similar directions in this high-dimensional space. "The contract terminates on December 31st" and "The agreement ends at year's close" will have vectors very close to each other, even though they share no words.

This is called **semantic similarity**. And it's what makes semantic search work. [6]

```python
from openai import OpenAI
client = OpenAI()

response = client.embeddings.create(
    input="The transformer architecture uses self-attention mechanisms.",
    model="text-embedding-3-large"
)

vector = response.data[0].embedding
print(len(vector))  # → 3072 floats
```

Those 3072 numbers encode the semantic content of that sentence. They capture which concepts are present, how they relate, what domain they belong to. A sentence about transformer attention will live in a completely different neighborhood of this 3072-dimensional space than a sentence about, say, contract law.

### Popular embedding models

- **OpenAI text-embedding-3-large** — 3072 dimensions, strong general performance
- **Cohere embed-v3** — strong multilingual support, good for production use
- **Nomic Embed** — open-source, great performance per token
- **E5-large-v2** — open-source, strong on retrieval tasks
- **BGE-M3** — multi-functionality: dense + sparse + multi-vector, multilingual [7]

The choice of embedding model affects everything downstream. Different models have different strengths across domains, languages, and query types. Testing multiple embedding models on your actual data is worth the effort before committing.

### Measuring similarity: cosine distance

Once everything is embedded, similarity is measured using **cosine similarity** — the cosine of the angle between two vectors. [6]

- **Score of 1.0** — vectors point in exactly the same direction. Semantically identical.
- **Score of 0.0** — vectors are orthogonal. No semantic relationship.
- **Score of -1.0** — vectors point in opposite directions. Opposite meanings.

Most real-world similar content scores between 0.7 and 0.95. The exact cutoff for "relevant" depends on your use case.

---

## Step 4: Vector Databases — The AI's Filing System

You've chunked your document into thousands of pieces. You've embedded each one into a vector. Where do all these vectors go?

Into a **vector database** — a purpose-built storage system that indexes vectors and lets you search them extremely fast.

The search isn't keyword matching. It's **approximate nearest neighbor (ANN) search**: given a query vector, find the stored vectors most similar to it. [8]

"Approximate" is doing work there. An exact nearest neighbor search over millions of vectors would take forever. ANN algorithms (HNSW, IVF, ANNOY) sacrifice a small amount of accuracy for enormous speed gains. A properly indexed vector database can search across tens of millions of vectors and return the top-K most similar in under 10 milliseconds.

### Popular vector databases

**Pinecone** — Fully managed, scales easily, great for production. Pay-per-use model.

**Weaviate** — Open-source, supports hybrid search (vector + keyword), highly configurable.

**Qdrant** — Written in Rust, extremely fast, strong filtering capabilities.

**ChromaDB** — Lightweight, great for prototyping and local development.

**FAISS** — Facebook's open-source library. Not a database per se, but excellent for local, high-performance vector search.

**pgvector** — Adds vector search to PostgreSQL. Useful when you want your vector store next to your relational data.

### Hybrid retrieval

Here's a practical point that trips up a lot of people. Dense vector search (semantic similarity) is great at finding conceptually related content, but it can miss exact matches for specific terms, proper nouns, model names, product codes, or technical jargon.

BM25 keyword search is the opposite — terrible at paraphrase, but great at exact term matching.

**Hybrid retrieval** combines both. You run a dense retrieval pass and a sparse (BM25) retrieval pass simultaneously, then merge the results using **Reciprocal Rank Fusion (RRF)** or a learned combiner. [9]

The combined approach consistently outperforms either method alone in production. Almost every serious RAG system uses hybrid retrieval.

---

## Step 5: Query Understanding — What Does the User Actually Want?

When a user sends a query, you can't always embed it as-is and expect great retrieval.

Short queries lack context. "What are the payment terms?" — payment terms for what? In which section? Sometimes users phrase questions very differently from how the answer is phrased in the document.

### Query expansion

Generate alternative phrasings of the query before retrieval. "What are the payment terms?" might expand to: "invoice schedule," "billing cycle," "payment due dates," "net 30 terms." Run retrieval on all of them, merge results.

### HyDE — Hypothetical Document Embeddings

A counterintuitive technique that works remarkably well. [10]

Instead of embedding the query directly, first ask the LLM to generate a *hypothetical answer* to the query — even if it's completely made up. Then embed that hypothetical answer, not the original query.

Why? Because the hypothetical answer reads like a document. It uses the vocabulary, phrasing, and structure of the document domain. Embedding it produces a vector much closer to where real answers live in the embedding space. Retrieval quality improves significantly, especially for complex questions.

```
Query: "What happens to the non-compete clause if I'm terminated for cause?"

Hypothetical answer (generated by LLM): 
"In the event of termination for cause, the non-compete restrictions 
in Section 4.2 remain fully enforceable for a period of 24 months..."

→ Embed this hypothetical, not the raw query
```

---

## Step 6: Retrieval and Reranking — Finding the Best Chunks

The query is embedded. The database is searched. Top-K chunks are returned.

But top-K from ANN search isn't the same as top-K in terms of actual relevance. ANN is fast but approximate. And the embedding comparison is a coarse measure.

This is where **reranking** comes in.

A reranker takes the top-K retrieved chunks (usually 20–50) and rescores them using a more accurate relevance model. The top-N after reranking (usually 3–10) get passed to the LLM.

### Why reranking is a different kind of model

Embedding models use a **bi-encoder** architecture: encode the query independently, encode the document independently, compare vectors. Fast because each encoding happens once. But because the query and document never directly interact during encoding, the relevance signal is coarser.

Rerankers use a **cross-encoder** architecture: the query and the document are concatenated and processed together in a single pass. The model can directly attend to relationships between query terms and document terms. The relevance signal is much richer. [11]

The tradeoff is speed. Cross-encoders are 10–50x slower than bi-encoders. Which is why you use bi-encoders to quickly narrow from millions to tens of candidates, then cross-encoders to precisely rank those tens.

Popular rerankers:
- **Cohere Rerank** — strong out-of-the-box, easy API
- **BGE-Reranker** — open-source, excellent performance
- **FlashRank** — lightweight, good for latency-sensitive applications

This two-stage architecture — fast approximate retrieval followed by precise reranking — is the production standard for high-accuracy RAG systems.

---

## Step 7: Generation — The LLM Finally Does Its Job

Top reranked chunks. User query. These get assembled into a prompt and passed to the LLM.

The prompt structure matters. A lot. The standard pattern:

```
System:
You are a precise document analyst. Answer the user's question 
using ONLY the information in the provided context. If the context 
does not contain the answer, say "Not found in the provided document." 
Do not draw on outside knowledge.

Context:
[CHUNK 1 — Source: contract.pdf, Page 14, Section 4.2]
"...the non-compete obligations shall remain enforceable for a period 
of twenty-four (24) months following the date of termination..."

[CHUNK 2 — Source: contract.pdf, Page 15, Section 4.3]
"...notwithstanding the above, termination for cause does not reduce 
the duration of non-compete restrictions..."

User question:
What are the non-compete terms if I'm terminated for cause?
```

The instruction "answer only from the context" is doing significant work here. It's a soft constraint that pushes the model away from confabulation — filling gaps with plausible-sounding but unverified information. It doesn't eliminate that tendency, but it significantly reduces it.

The LLM then generates an answer. Because the actual relevant text is right there in front of it, the answer is grounded, accurate, and citable.

### Citation and attribution

The best production systems don't just return answers — they return **grounded answers with source attribution**. Every claim maps back to a specific chunk, which maps back to a specific page, section, and document.

This transforms the AI from something you use casually to something you can rely on professionally. In legal, medical, financial, and regulatory contexts, this isn't a nice-to-have. It's the whole point.

---

## Where It Still Breaks

Here's the honest part. Even with all of this — parsing, chunking, embedding, hybrid retrieval, reranking, grounded generation — things still go wrong. Regularly.

### Retrieval miss

The relevant chunk wasn't retrieved. The user's query phrasing and the document's phrasing were semantically far apart in embedding space. BM25 didn't catch the exact terms either. The model generates an answer anyway — from wrong or incomplete context. The answer sounds confident.

This is the most common failure mode and the hardest to detect, because you only know it happened if you also know the right answer.

### Lost context at chunk boundaries

A critical piece of information spans the boundary between two chunks. Neither chunk contains the complete thought. Both get retrieved. But the model can't stitch them together correctly without the missing context from the overlap.

This is why overlapping chunks and semantic-aware chunking strategies matter so much.

### Model confabulation from thin context

The retrieved context is technically relevant but doesn't contain enough detail to answer the question fully. The model, deeply conditioned to be helpful and produce coherent text, fills the gap — from its training data, from inference, from analogy. The answer sounds plausible. It might even be partially correct. But it's not grounded in the document.

### Parsing corruption

A PDF was poorly extracted. An equation was garbled. A table got scrambled. These errors create bad chunks, which create bad embeddings, which either miss retrieval entirely or land the wrong content in front of the model. The corruption is silent throughout the pipeline and emerges as a confident wrong answer at the end.

### Evaluating RAG quality

You need to measure these things, not just test them manually. The standard framework is **RAGAS** — Retrieval-Augmented Generation Assessment Score — which measures: [12]

- **Faithfulness** — does the generated answer reflect what's actually in the retrieved context?
- **Answer relevancy** — is the answer on-topic with the question asked?
- **Context precision** — are the retrieved chunks actually relevant?
- **Context recall** — did the retrieval capture all the necessary information?

Running automated evals like RAGAS over representative query sets is now standard practice in teams building production document AI.

---

## Advanced Patterns Worth Knowing

### Multi-hop retrieval

Some questions can't be answered in a single retrieval step.

"What does the leave policy say about contractors who've been employed for less than six months?" — you might need to retrieve the leave policy, then the contractor classification definition, then the six-month probation terms, and combine all three.

Multi-hop retrieval chains multiple retrieve → read → re-query cycles. Each step uses information from the previous retrieval to formulate the next query. Think of it as the AI needing to look up a word, then follow a cross-reference, then check a footnote before it can answer you.

### Agentic RAG

Instead of a fixed retrieval pipeline, an AI agent dynamically decides how to retrieve. It retrieves, evaluates whether the context is sufficient, reformulates the query if not, retrieves again, synthesizes across multiple retrieval results, and calls external tools if needed. [13]

Frameworks like LangGraph and AutoGen make this kind of orchestration possible. The quality can be significantly higher than static RAG — at the cost of latency and added complexity.

### Long-context vs. RAG: the real answer

As context windows grow to 200K, 1M tokens, people ask: does RAG even matter anymore? Can you just dump everything in?

The honest answer: both approaches work, for different scenarios.

Long-context (full-document) works well for:
- Fixed, well-scoped documents (a single contract, a specific technical spec)
- Questions that require synthesizing information across the whole document
- Cases where you want maximum recall and can afford the cost

RAG works better for:
- Large, dynamic document corpora (thousands of documents, updated regularly)
- Latency-sensitive applications (you don't want to process 1M tokens per query)
- Auditability requirements (trace exactly what was used to answer)
- Cost-sensitive deployments

The best modern systems often use both together. Store documents in a RAG system. For complex synthesis queries, retrieve a broad context, combine the relevant chunks into a long-context prompt, and generate. The strengths compound.

---

## The Full Stack, Named

Here's what the complete modern pipeline looks like in terms of actual tools:

**Ingestion/Parsing:** LlamaParser, Unstructured.io, Amazon Textract, Azure Document Intelligence

**Chunking:** LangChain's `RecursiveCharacterTextSplitter`, LlamaIndex `SentenceSplitter`, custom semantic chunkers

**Embedding:** OpenAI text-embedding-3-large, Cohere embed-v3, Nomic Embed, BGE-M3

**Vector Store:** Pinecone, Weaviate, Qdrant, ChromaDB (dev), pgvector (Postgres users)

**Hybrid Search:** Weaviate (built-in BM25), Qdrant sparse vectors, Elasticsearch

**Reranking:** Cohere Rerank, BGE-Reranker, FlashRank

**Orchestration:** LangChain, LlamaIndex, Haystack

**Evaluation:** RAGAS, TruLens, DeepEval

**LLM Generation:** GPT-4, Claude 3, Gemini 1.5 Pro, Mistral, Llama 3

A year ago, assembling this stack took weeks. Today, frameworks abstract most of the plumbing. You can have a basic working pipeline in a day. What takes time now is *tuning* it — chunk size experiments, embedding model comparisons, retrieval threshold calibration, eval set construction. That's where the real work lives.

---

## What This All Means, Actually

Here's the perceptual point. When AI reads your document, it's not reading the way you read. It's not understanding the way you understand.

It's running a pipeline. Each stage transforms the document into a form the next stage can use. Parse. Chunk. Embed. Store. Retrieve. Rerank. Generate. Seven distinct operations, each one a potential failure point, each one a lever you can tune.

The reason modern document AI feels capable — genuinely capable — is that every one of those stages has gotten dramatically better in the past two years. Better parsers. Better embedding models with more semantic precision. Better vector databases with smarter indexing. Better rerankers. Better prompting techniques for grounded generation. The quality you see is the compounding effect of improvements across the entire stack.

The reason it still fails — confidently, sometimes embarrassingly — is that each stage can silently drop information. And the model generating the final answer has no direct way to know what was missed. It only knows what it was given.

This is why grounding matters. Why attribution matters. Why evaluation pipelines matter. The best document AI systems don't just produce answers — they produce answers you can verify, trace, and trust.

The AI that knows where its answer came from is more useful than the one that sounds most confident.

---

## Outro: The Shape of What's Coming

The field isn't stopping here.

Context windows are going to keep growing. Attention mechanisms are getting more efficient — with architectures like sliding window attention, sparse transformers, and mixture-of-experts making very long contexts more tractable computationally. [14][15]

Embedding models are getting smarter at capturing domain-specific semantics. Retrieval is becoming agentic — dynamic, multi-step, adaptive to what the model needs. Evaluation frameworks are maturing to the point where you can actually measure whether your system improved.

The trajectory is clear: AI that can reason reliably over *any* document, at any scale, with full auditability. We're not there yet. But the gap between where we are and where we need to be is measured in engineering effort, not fundamental impossibility.

The pipeline is the foundation. Understanding it isn't optional for anyone building in this space. It's the thing you come back to when something breaks — and something always breaks.

Know the pipeline. Know where it can fail. Build accordingly.

---

## References

[1] Vaswani, A., et al. "Attention Is All You Need." *NeurIPS 2017.* arXiv:1706.03762

[2] Liu, N. F., et al. "Lost in the Middle: How Language Models Use Long Contexts." *arXiv:2307.03172*, 2023.

[3] Lewis, P., et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." *NeurIPS 2020.* arXiv:2005.11401

[4] Smith, R. "An Overview of the Tesseract OCR Engine." *ICDAR 2007.*

[5] LangChain Documentation. "Text Splitters — RecursiveCharacterTextSplitter." docs.langchain.com

[6] Reimers, N. & Gurevych, I. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." *EMNLP 2019.* arXiv:1908.10084

[7] Chen, J., et al. "BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation." *arXiv:2402.03216*, 2024.

[8] Johnson, J., Douze, M. & Jégou, H. "Billion-scale similarity search with GPUs." *arXiv:1702.08734*, 2017.

[9] Cormack, G. V., Clarke, C. L. A., & Buettcher, S. "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods." *SIGIR 2009.*

[10] Gao, L., et al. "Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE)." *arXiv:2212.10496*, 2022.

[11] Nogueira, R. & Cho, K. "Passage Re-ranking with BERT." *arXiv:1901.04085*, 2019.

[12] Es, S., et al. "RAGAS: Automated Evaluation of Retrieval Augmented Generation." *arXiv:2309.15217*, 2023.

[13] Shinn, N., et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." *arXiv:2303.11366*, 2023.

[14] Su, J., et al. "RoFormer: Enhanced Transformer with Rotary Position Embedding." *arXiv:2104.09864*, 2021.

[15] Press, O., et al. "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (ALiBi)." *arXiv:2108.12409*, 2021.

---

*Written by Abhishek Shah*
*abhishekshah.vercel.app · abhishek.aimarine@gmail.com*

theindexproject

theindexproject Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

theindexproject

theindexproject Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README