What Is RAG and Why Does It Eliminate Hallucination?
Retrieval-Augmented Generation (RAG) is an architecture pattern that grounds an LLM's responses in your actual data rather than whatever it picked up during training. A user asks a question. The system goes and searches your documents for the relevant bits, then hands that context to the LLM alongside the question. So the answer comes from YOUR data, not the model's general knowledge of the world.
Without RAG, LLMs hallucinate. They hand you answers that sound right and are flat-out wrong. Ask one about your product pricing or a company policy or where a customer's order is, and it has nothing to go on. None of that lived in its training data.
Add RAG and hallucination drops off a cliff. The model still writes the language. It just writes it while staring at the real source material. And if the answer is not in the context it pulled back, a RAG system worth its salt will say 'I don't have information about that' instead of inventing something.
Where does this show up in practice? Support bots that answer straight from your knowledge base. Internal tools that search company policy. Legal research that cites the exact document it pulled from. Product assistants that actually know your feature set. Our Python development team builds these RAG systems for production, and our AI/ML engineers own the pipeline end to end.
How Does RAG Architecture Work?
A RAG pipeline runs in four stages. Here is how each one earns its keep.
Stage 1, Document Processing: Pull in your source documents. PDFs, web pages, Notion docs, API references, whatever you have. Clean the text. Strip the headers, footers, and nav junk. Then split it into chunks (more on that below). People underrate this step constantly, and they pay for it later. Garbage in, garbage out.
Stage 2, Embedding: Turn each chunk into a vector, which is really just a list of numbers that captures what the text means. You run it through an embedding model. We reach for OpenAI text-embedding-3-small, which bills a couple of cents per million tokens, Cohere embed-v3, or an open-source option like Sentence Transformers when we want to self-host. Every chunk ends up as a point in vector space, and similar content lands close together.
Stage 3, Retrieval: A question comes in. Embed it with the same model you used on your documents (this matters, mismatched models retrieve garbage). Then search the vector database for the top-K closest chunks, usually K=3 to 5. Those chunks are your retrieved context. That is the data the LLM gets to answer from.
Stage 4, Generation: Now you build the prompt. System instructions, then the retrieved context, then the user's question. Send all of it to an LLM (we tend to use Claude Sonnet or GPT-4o). The model writes an answer grounded in those chunks. One thing we never skip: attach the sources, so a user can check the answer against the document it came from.
Latency on a RAG system that is built well runs 1 to 3 seconds end to end. Embedding is roughly 50ms, retrieval 100 to 200ms, and generation 1 to 2.5 seconds once you stream the tokens back. Get a free RAG architecture consultation.
Which Vector Database Should You Use?
Pinecone: Fully managed, basically zero ops. The Starter plan covers 100K vectors on a flat monthly fee. Great fit if your team does not want to babysit infrastructure. The catch? You are locked into the vendor, the price climbs with vector count rather than queries, and there is no self-hosting escape hatch.
Weaviate: Open source and self-hostable. Runs on Docker or Kubernetes. It ships with hybrid search baked in (vector plus keyword), which is genuinely handy. It is more work to operate than Pinecone, but nobody owns you. Free if you host it yourself. Weaviate Cloud is there if you would rather not run it yourself.
pgvector: A PostgreSQL extension. It lives inside the Postgres database you already run, so there is no new infrastructure to stand up. On Supabase it comes built in. Performance holds up nicely to about 1 million vectors. Past that point, a dedicated vector database starts pulling ahead.
What we usually do: If you are already on PostgreSQL or Supabase, start with pgvector. It is free and you add nothing to your stack. Graduate to Pinecone or Weaviate once you blow past a million vectors, or once you need sub-50ms retrieval at real scale. No sooner. Switching early is just busywork.
Here is the cost side by side for 500K vectors and 10K daily queries. pgvector on Supabase Pro is the cheapest all in. Pinecone Starter runs nearly three times that, Weaviate Cloud about double. Self-hosted Weaviate on a DigitalOcean box matches pgvector, but you own the ops.
What Chunking Strategies Actually Work for RAG?
Honestly, how you chunk your documents decides RAG quality more than which model or vector database you pick. We have watched a bad chunking pass tank an otherwise solid system. Three strategies are worth knowing.
Fixed-size chunking: Split every N tokens, usually 256 to 512. Dead simple to implement. It does fine on uniform content like blog posts and articles. It falls apart on structured documents, where a blind split can slice a table or a code block clean in half.
Semantic chunking: Split at the natural seams instead. Paragraph breaks. Section headings. The end of a sentence. Each chunk keeps its meaning intact. It takes more effort to build, since you need some NLP to find sentence boundaries reliably. It pays off most on documentation, policy docs, and technical guides.
Hierarchical chunking: Set up parent and child chunks. The parent holds a whole section. The children hold the individual paragraphs inside it. When a child chunk matches a query, you pull the parent back too, so the LLM sees the full context around the match. For anything with nested structure, like legal contracts, technical specs, or API docs, this is the strategy that wins.
Chunk size matters more than people expect. Go too small (100 tokens) and each chunk is starved of context, so retrieval gets noisy. Go too large (2000 tokens) and the chunk drowns the actual answer while burning context window. For most cases the sweet spot is 300-500 tokens with 50-token overlap between neighboring chunks.
Metadata enrichment: Tag every chunk with its source URL, document title, section heading, and date. That extra metadata lets you filter (search product docs only, leave the blog posts out) and lets you cite the source right in the answer.
How Do You Measure RAG Pipeline Quality?
You cannot improve what you do not measure. With RAG, three metrics carry the weight.
Faithfulness: Does the answer actually match the context it was given? Scored 0 to 1. A faithful answer only claims things the retrieved chunks support. An unfaithful one sneaks in facts from the model's training data, which is hallucination by another name. We aim for above 0.85.
Relevancy: Did retrieval grab the right chunks in the first place? Scored 0 to 1. You check whether the chunks it pulled actually hold the information the question needs. Feed an LLM the wrong chunks and even the best model gives you a weak answer. We target above 0.80.
Answer Correctness: Is the final answer right and complete, judged against a ground-truth set? This one needs a hand-curated dataset of 50 to 100 question-answer pairs. It is a pain to build, no way around that. For anything going to production, it is non-negotiable.
How we wire up evaluation: RAGAS (open source, Python) handles the automated scoring. Build a test set of 50 questions whose answers you already know. Re-run it every week, and always after you touch chunking, prompts, or retrieval settings. We treat these scores like unit tests. If a number drops, something broke, and you go find out what.
Why Does a RAG System That Demoed Perfectly Fail in Production?
It answers confidently and it is wrong, and it is wrong in a way that reads exactly like being right. Support has quietly stopped trusting it. Somebody has already suggested moving to a bigger model.
Hold off on that. A bigger model fixes one of the four things that commonly break here, and it is not usually the one that is broken. The demo was not a lie either. It ran on a different corpus, and that is the whole story.
1. The corpus got big and got contradictory. The demo indexed 40 documents somebody chose. Production indexes 40,000 that nobody chose, and among them are three versions of the same policy written in 2023, 2024 and 2026. All three are semantically close to the question. Retrieval has no idea which one is current, because recency is metadata and similarity search does not read metadata unless you told it to. The model then blends all three into one fluent, wrong paragraph. This is the failure mode nobody plans for and it arrives the week you finish the migration.
2. Real questions do not look like test questions. The evaluation set was written by people who know the documents. Real users ask things like 'why did mine get rejected' with no nouns in the sentence at all. Embedding a question with no content words retrieves noise, and the top-k comes back full of plausible neighbours. Retrieval never fails loudly here. It always returns something.
3. The chunk that holds the answer is not the chunk that matches the question. Documents put the question language in a heading and the answer several paragraphs down, so the chunk scoring highest is the one describing the problem rather than the one resolving it. You can see this instantly by reading what got retrieved, and almost never by reading the answer.
4. The model answered from training data. Retrieval returned nothing useful, and rather than saying so the model wrote a reasonable answer out of what it already knew. Fluent, general, undated, and often close enough to pass a skim. This is the one a stronger model genuinely helps with, and it is also the one an explicit refusal path helps with more.
The reason teams thrash here is that all four look identical from the outside. Same symptom, four different fixes, and picking the wrong one costs a sprint. So before changing anything, find out which half of the pipeline is at fault.
Related: if your pipeline retrieves anything a person outside your organisation could have written, that content can also carry instructions rather than facts, which is a separate failure with a separate fix. See prompt injection in production AI agents.
How Do You Tell a Retrieval Failure From a Generation Failure?
Log the retrieved chunks alongside every answer, then read them. That is the whole diagnostic, and it is skipped constantly because the answer is the interesting part and the chunks are not.
The rule is short. If the answer was in the retrieved chunks and the model got it wrong, that is generation. If the answer was not in the chunks at all, that is retrieval, and no prompt change will save you. Teams spend weeks rewriting system prompts against what turns out to be a retrieval problem, because the prompt is the easiest thing in the system to edit.
Build the ground-truth set out of production traffic instead of waiting for a perfect one. The evaluation section above is right that you need 50 to 100 question-answer pairs and right that building them is tedious. The shortcut is that you do not have to invent them. Take the real queries that collected a thumbs down, have somebody who knows the documents record which chunk should have been returned, and you have a labelled retrieval set drawn from the distribution you actually serve. Fifty real failures beat two hundred invented questions, because the invented ones share the blind spot of the person who wrote them.
Then measure retrieval on its own terms, separately from the answer. Recall at k asks whether the correct chunk appeared anywhere in the top k, and it is the number that tells you whether generation ever had a chance. Mean reciprocal rank asks how near the top it landed, which matters because a correct chunk sitting at position nine competes with eight distractors for the model's attention. Track the empty-retrieval rate too, meaning queries where nothing crossed your similarity threshold, and if that number is zero you do not have a threshold, you have a top-k.
What each fix is actually for:
Better chunking when the right document is retrieved but the specific passage is not. Hybrid search when failures cluster on product names, error codes, SKUs and other exact strings that embeddings smear together. Metadata filtering when the corpus contains multiple versions or tenants, which is the contradiction failure above and the one filtering solves outright. Reranking when recall at 20 is strong but recall at 3 is weak, meaning the right chunk is being found and then buried. A stronger model or a refusal path when the chunks were right and the answer still was not.
On reranking specifically, because it is the fashionable answer. A cross-encoder rescoring the top 20 down to the top 3 does improve ordering, and it also adds a network hop, latency and a second model to keep working. It is a fix for a ranking problem. If recall at 20 is poor, reranking has nothing good to promote and you have added a moving part to a pipeline that was already broken further upstream. Check recall first.
And give the system a way to say no. A retrieval score threshold below which the pipeline answers that it does not know, with a link to a human, converts the worst failure mode into the second worst. Users forgive an assistant that admits ignorance. They stop using one that was confidently wrong twice.
What Are the Best RAG Deployment Patterns?
Cache the questions everyone asks: When 100 people all ask 'What is your refund policy?', running the full pipeline 100 times is just wasted money. Cache the question embeddings and their answers instead. A plain Redis cache with a one-hour TTL has cut our costs 40 to 60% on customer-facing apps.
Hybrid search: Run vector similarity and keyword search (BM25) together. Vector search is great at finding content that means the same thing. Keyword search nails the exact matches that vectors fumble, like product names, error codes, and IDs. Put them together and retrieval relevancy beats either one alone by 15 to 25%.
Feedback loops: Put a thumbs up and thumbs down on every answer. Watch which queries collect the thumbs down. That list is your to-do list. Sit with those conversations once a week and the pattern usually shows itself. Sometimes the documentation is just missing, so you write it. Sometimes the chunking was bad, so you re-chunk that one doc. Sometimes the system prompt needs a tweak.
Monitoring: Log all of it. Every query, the chunks it retrieved, the answer it produced. Then keep an eye on the numbers that matter. Retrieval latency (we aim under 200ms). Generation latency (under 3s). The empty-retrieval rate, meaning queries that come back with nothing relevant. And a user satisfaction score on top.
What a production RAG system costs: a multi-month build, then a standing monthly line covering the embedding API, the LLM API, vector DB hosting, and app hosting. One thing worth knowing: cost tracks your document volume and query volume, not how many users you have.
Related: AI agents architecture
