RAG Architecture: Making AI Work on Your Data
RAG Architecture: Making AI Work on Your Data
TL;DR: A large language model knows a lot about the world and nothing about your business. Retrieval-augmented generation — RAG — is how you change that. It connects a model to your documents, your knowledge base, your CRM records, and lets it answer questions with real information instead of general knowledge. But building a RAG system that actually works is harder than the tutorials suggest. This article covers the full pipeline, the failure modes that quietly undermine output quality, and the evaluation discipline that turns a prototype into a production system you can trust.
Introduction
There is a moment in every AI project where the limitations of a generic model become impossible to ignore.
You ask it a question about your product. It gives a confident answer that sounds right but is completely wrong — because it was trained on outdated public data from two years ago. You ask it to summarise your internal policy. It makes reasonable inferences that contradict your actual policy, because it has never seen the document. You ask it about a customer. It hallucinates a relationship that does not exist.
This is not a model quality problem. It is a knowledge access problem. The model is capable of reasoning — it just does not know the facts that matter to your business. And the only way to give it those facts is to connect it to your data.
That is what retrieval-augmented generation does. RAG is the bridge between a general-purpose language model and the specific information your business runs on. It lets the model retrieve relevant documents, passages, or records at query time and use them as grounding context for its answer. The model is no longer guessing — it is working from source material.
It sounds straightforward. In practice, building a RAG system that produces reliable, accurate, and useful answers is a systems engineering challenge that trips up even experienced teams. The pipeline has five stages, and a weakness in any one of them propagates through to the output. This article walks through each stage, the failure modes to watch for, and the evaluation practice that separates a demo from a production system.
What RAG actually is (and what it isn't)
RAG is not a product. It is not a feature you toggle on. It is an architecture — a pattern for combining retrieval systems with generation systems so the model works with information it was not trained on.
Here is what RAG is. A user asks a question. The system converts that question into a search query and looks for relevant material in a connected knowledge source — a vector database of your documents, a search index of your knowledge base, or a set of APIs that pull records from your business systems. It retrieves the most relevant results, packages them into a prompt alongside the user's question, and sends the whole thing to the model. The model reads the retrieved context, understands it, and produces an answer grounded in that context. If the retrieval is good, the answer is accurate. If the retrieval is weak, the answer is guesswork dressed up as fact.
Here is what RAG is not. It is not the same as fine-tuning — teaching the model new facts permanently by retraining it. Fine-tuning changes what the model knows. RAG changes what the model can access. The two techniques are complementary, and most production systems use both, but they solve different problems. RAG is also not a search engine with a summary on top. A search engine returns links. A RAG system reads retrieved content, synthesises across multiple sources, resolves conflicts, and produces an answer that may not exist in any single source document. The generation step is doing real cognitive work.
The architecture matters because the expectations matter. If you think of RAG as a search plugin, you will underinvest in retrieval quality and blame the model when answers are wrong. If you think of it as an AI system that happens to look things up, you will underinvest in the retrieval pipeline and blame the model again. RAG is a joint system. Both sides need to work.
The RAG pipeline: five stages from document to answer
Stage 1: Ingestion
Ingestion is the process of getting your data into a form the retrieval system can work with. It starts with a simple question: where does the knowledge live?
The answer is usually scattered. A portion of it is in structured formats — database records, CRM fields, product catalog entries. A larger portion is unstructured — PDFs, Word documents, wiki pages, support tickets, meeting transcripts, email threads. Some of it has never been written down at all — it lives in team channels, Slack threads, and the institutional memory of long-tenured employees.
The ingestion stage is about inventorying these sources, extracting the text from them, and preparing it for processing. For structured data, this means building connectors that can pull records on a schedule. For unstructured documents, it means parsing — extracting clean text from PDFs, stripping formatting artefacts, handling tables and images that contain important information. For real-time sources like Slack or support channels, it means deciding what to index and what to leave out — because not every message is useful context for an AI system.
The output of ingestion is a clean corpus of documents. If a document is important but cannot be cleanly extracted, the pipeline is already compromised before any retrieval happens.
Stage 2: Chunking
Chunking is the least glamorous stage of the pipeline and the one most responsible for retrieval failures in production. The problem is straightforward: documents are long, and retrieval systems work best with short passages. You need to split documents into chunks that are small enough to be matched precisely but large enough to carry enough context that the model can understand them.
The simplest approach is fixed-size chunking — split every document into 500-token chunks with some overlap. It works. It also produces chunks that cut off mid-sentence, split a heading from its body, and separate a question from its answer. The model receives a chunk that says "…the third quarter results showed a decline, driven primarily by" and has no idea what the decline was driven by, because that was in the next chunk.
Better approaches account for document structure. Semantic chunking uses the model itself to detect natural topic boundaries and split at meaningful transitions. Hierarchical chunking preserves the document structure — headings, subheadings, sections — so each chunk carries its position in the larger document. For structured data like product catalogs, entity-aware chunking keeps related fields together so the chunk for a product includes its name, description, price, and specifications as a unit.
Chunk size is a trade-off. Larger chunks preserve more context but dilute retrieval precision — the system retrieves a 2,000-word passage when only one paragraph is relevant, and the model has to sift through noise. Smaller chunks are more precise but lose context — the model sees a paragraph in isolation and misses the section that explains what it means. The right size depends on the content, the use case, and the expected questions. A support knowledge base works well with 300–500 token chunks. A legal document with dense cross-references may need larger chunks with generous overlap.
Stage 3: Embedding
Once documents are chunked, each chunk needs to be converted into a vector — a numerical representation that captures its semantic meaning. This is done by an embedding model, which reads a chunk of text and produces a list of numbers that represent what the text is about. Chunks with similar meanings get similar vectors, even if they use different words.
The embedding model you choose has a direct impact on retrieval quality. General-purpose embedding models like OpenAI's text-embedding-3 or open-source alternatives like bge-large-en work well for most business content. But if your domain is specialised — medical, legal, highly technical — a domain-specific embedding model will produce vectors that capture the nuances of your vocabulary better.
The embedding model also determines the dimensionality of your vectors — typically 768, 1024, or 1536 dimensions. Higher dimensionality can capture more semantic detail but increases storage and search costs. For most business RAG systems, 768 or 1024 dimensions provide a good balance between accuracy and efficiency. The vectors are stored in a vector database — Pinecone, Weaviate, Qdrant, or pgvector on top of PostgreSQL — where they can be searched at query time.
A common mistake is embedding documents and then never re-embedding them. Your documents change. New information arrives. Old information becomes stale. A RAG system that embeds once and never updates is a system that gets progressively less accurate. The embedding stage needs a refresh schedule — nightly, weekly, or real-time, depending on how fast your knowledge base changes.
Stage 4: Retrieval
Retrieval is where the user's question meets your indexed knowledge. The user's query is embedded using the same model that embedded the documents. The vector database finds the chunks whose vectors are closest to the query vector — the ones that are semantically most similar.
This is called semantic search, and it is powerful. It finds documents that are relevant even when the user uses different words than the document. A query for "how do I reset a customer's password" can retrieve a chunk titled "account recovery procedure" because the embedding model understands they are about the same thing.
But semantic search alone has blind spots. It can miss exact matches for specific terms — a query for "policy B-203" might not retrieve the document titled "B-203 Policy" if the embedding model does not treat the policy number as a distinct entity. It can also be fooled by documents that are semantically similar but factually irrelevant — a general explanation of password security when the user specifically needs the reset procedure.
This is why most production RAG systems use hybrid search: semantic search combined with keyword search. The keyword layer catches exact matches — policy numbers, product codes, proper names — that semantic search might miss. The semantic layer catches conceptual matches that keyword search would miss. The results are combined and ranked to produce a final set of retrieved chunks.
The retrieval stage also includes a reranking step in more advanced systems. A initial retrieval pulls a larger set of candidates — say, 50 chunks — and then a more powerful but slower model reranks them, selecting the top 5 or 10 that are most relevant to the specific query. This two-stage approach gives you the speed of vector search with the precision of a more sophisticated relevance model.
Stage 5: Generation
The final stage is where the model reads the retrieved context and produces an answer. This is more than just summarisation. The model needs to understand the question, evaluate the retrieved chunks — some of which may be partially relevant or even contradictory — synthesise across them, and produce a coherent answer that is grounded in the provided context.
The generation prompt matters enormously. A good RAG prompt does three things. First, it instructs the model to answer only from the provided context and to indicate clearly when the context does not contain the answer. This prevents the model from falling back on its general knowledge when retrieval is weak — which is exactly when you do not want it improvising. Second, it includes citations or references so the user can verify the answer against the source material. Third, it handles edge cases gracefully — instructing the model what to do when the retrieved context is contradictory, incomplete, or clearly irrelevant to the question.
A well-designed generation stage is the difference between a system that says "I don't know" when appropriate and one that confidently makes things up. The first earns trust. The second erodes it.
Why retrieval quality determines output quality
There is a simple rule in RAG systems: the model cannot produce a good answer from bad context. It does not matter how capable the generation model is — if the retrieval stage returns the wrong chunks, the output will be wrong. If it returns partially relevant chunks, the output will be plausible but imprecise. If it returns nothing, the model either admits it does not know — which is honest but unhelpful — or it falls back on its general knowledge and hallucinates.
This means the retrieval stage is the most important component in a RAG system. It is also the hardest to get right, because retrieval quality depends on every upstream stage. If ingestion misses a document, it cannot be retrieved. If chunking splits a key passage across two chunks, neither chunk has enough context to be useful. If the embedding model does not understand your domain's vocabulary, relevant documents will not match the query. If the retrieval strategy does not handle the types of questions users actually ask, it will return the wrong information.
The practical implication is that RAG evaluation has to start at the retrieval layer. Before you evaluate whether the model's answers are good, evaluate whether the right chunks are being retrieved. If retrieval is weak, no amount of prompt engineering at the generation stage will fix it. Most RAG projects that underperform are not suffering from a model problem. They are suffering from a retrieval problem that nobody diagnosed.
Common failure modes in production RAG systems
After building and debugging RAG systems across multiple domains, the same failure patterns appear repeatedly. They are worth naming because they are predictable — and preventable — once you know to look for them.
Chunking that destroys context. A policy document says "exceptions to the above rule include the following cases." The chunk contains only this sentence, because the chunker split at a paragraph boundary. The model reads it and asks: what rule? What cases? Neither is in the chunk, and the answer is useless. The fix is overlap — ensuring that chunk boundaries include enough surrounding text that cross-references and dependencies are preserved.
Stale embeddings. The knowledge base was embedded three months ago. Since then, the product catalog has changed, three policies were updated, and a new product was launched. Queries about the new product return nothing useful. Queries about the updated policies return the old version that is no longer accurate. The fix is an embedding refresh pipeline that runs on a schedule matching the rate of change in your content — or triggers automatically when source documents are updated.
Context window overload. The retrieval system pulls 15 chunks, each 1,000 tokens long, and feeds all of them into the generation prompt. The model's context window is 8,000 tokens. By the time the prompt instructions, the chat history, and the retrieved chunks are combined, the model is over its limit. The most relevant chunks are truncated, and the model answers from whatever happened to survive. The fix is retrieval discipline — retrieving fewer chunks of higher quality, using reranking to prioritise the most relevant ones, and respecting the model's context window as a hard constraint.
The empty retrieval trap. A user asks a question using terminology that does not appear in any document. The embedding model does not understand the mapping between the user's language and the document's language. Retrieval returns nothing, and the model — if not instructed otherwise — improvises. The fix is a retrieval quality threshold: if the similarity scores of the retrieved chunks are below a minimum, the system should tell the user it cannot find relevant information rather than attempting to generate an answer from nothing.
Query-document mismatch. A user asks "what is our refund policy for international customers." The chunk for the refund policy was indexed. The chunk for the international shipping policy was indexed. But no chunk covers the intersection — what happens when both conditions apply. The retrieval system returns both, and the model tries to synthesise across them, producing an answer that is not actually supported by either document. The fix is to test retrieval against the questions users actually ask and identify gaps in how the content is structured and chunked before it reaches production.
Hybrid search: keyword and semantic, together
Semantic search alone is not enough. Neither is keyword search. The two approaches have complementary strengths, and the best production RAG systems use both.
Semantic search excels at understanding intent. It finds the document about "employee departure procedures" when the user searches for "offboarding checklist." It finds the support article about "payment processing delays" when the user asks "why hasn't my transaction gone through." It handles the natural variation in how different people ask the same question.
Keyword search excels at precision. It finds the exact document titled "Policy B-203" when the user searches for "B-203." It returns the product with SKU "AK-447" when the user types that exact code. It catches named entities, identifiers, and specific terminology that embedding models, which see words as continuous semantic signals, can blur.
A hybrid search system runs both queries in parallel. The semantic query returns chunks ranked by embedding similarity. The keyword query returns chunks ranked by lexical match score — BM25, the standard algorithm, or a more modern variant. The results are merged and reranked. The simplest fusion strategy is reciprocal rank fusion, which combines rankings rather than raw scores — but more sophisticated approaches use a learned model to weight the contributions based on the query type.
The result is a retrieval layer that handles both "find me everything about our return policy" (semantic) and "find document HR-2026-03" (keyword) with equal reliability. For most business RAG systems, hybrid search is not an optimisation — it is a baseline requirement for production readiness.
Evaluating and improving RAG performance over time
RAG systems degrade. Documents change, user questions evolve, and edge cases emerge that were not anticipated during development. A RAG system that is not evaluated regularly will, over time, produce answers that are less accurate, less relevant, and less useful — and nobody will notice until a wrong answer causes a visible problem.
Evaluation starts with retrieval metrics. Before you assess whether the model's answer is good, assess whether the right chunks were retrieved. The standard metric is recall: for a given question, what percentage of the relevant documents were retrieved? If recall is low, the model never saw the information it needed, and no amount of generation quality can compensate. Mean reciprocal rank — how high the first relevant chunk appears in the results — tells you whether the system prioritises well.
Generation metrics come next. Does the answer correctly reflect the retrieved context? Is it complete — covering all the information the user needs? Is it faithful — not adding claims from the model's general knowledge that are absent from the context? These are harder to measure automatically, but human evaluation on a sample of queries provides a signal. For higher-volume systems, using a separate LLM as an evaluator — asking it to score answers against the retrieved context and a reference answer — can scale the evaluation process.
The improvement loop is straightforward. Log a sample of real user queries. Evaluate retrieval quality on that sample. Identify the queries where retrieval failed. Diagnose why — was the relevant document missing from the index? Was it chunked poorly? Did the embedding model not recognise the query-document relationship? Fix the root cause, re-index, and retest. This is not a one-time exercise — it is a continuous discipline that keeps the RAG system honest.
The businesses that get the most value from RAG are not the ones that built the most sophisticated pipeline on day one. They are the ones that built a solid pipeline and then invested in the evaluation and improvement cycle that makes it better every month. RAG is not a build-once system. It is a living system that rewards attention.
FAQs: RAG Architecture
Do we need RAG if we are already fine-tuning our model?
Yes. Fine-tuning and RAG solve different problems. Fine-tuning changes the model's behaviour — its tone, its reasoning patterns, its domain fluency. It is good for teaching the model how to think about your business. RAG gives the model access to facts that change regularly — product catalogs, policies, documentation, customer records. You do not want to retrain a model every time your product pricing changes. RAG handles that at query time. Most production systems use both: a fine-tuned model with a RAG pipeline for fresh, specific information.
How do we choose a vector database?
For most business RAG systems, the choice depends on your existing infrastructure. If you already run PostgreSQL, pgvector is the simplest path — it adds vector search to the database you already know, without introducing a new system to manage. If you need higher scale, lower latency, or advanced features like hybrid search and filtering, dedicated vector databases like Pinecone, Weaviate, or Qdrant provide purpose-built tooling. Start with the simplest option that meets your requirements. You can migrate later if you outgrow it — the embeddings are portable.
How often should we re-embed our documents?
It depends on how fast your content changes. A product catalog with weekly price updates should be re-embedded weekly. A knowledge base that gets a few edits per month can be re-embedded monthly. Real-time systems — support chatbots that need to know about a policy change the moment it is published — need event-driven re-embedding that triggers when source documents are modified. The rule of thumb: re-embed at least as often as your most frequent content change.
What is the biggest mistake teams make with RAG?
Treating RAG as a model integration rather than a data engineering problem. The model is the easy part. The hard part is the ingestion pipeline that extracts clean text from messy sources, the chunking strategy that preserves context, the embedding model that understands your domain, and the retrieval layer that reliably finds the right information. Teams that invest engineering time in the data pipeline get reliable RAG. Teams that focus on the model and rush the pipeline get a demo that does not survive contact with real users.
Can we just use a RAG-as-a-service platform?
Yes — for many use cases, managed RAG platforms like OpenAI's Assistants API, Anthropic's tool use, or dedicated services like LlamaIndex Cloud handle the pipeline for you. The trade-off is control and customisation. A managed platform abstracts away chunking, embedding, and retrieval — which saves engineering time but limits your ability to tune each stage for your specific content and user questions. For simple use cases, a managed platform is a pragmatic choice. For complex use cases with specialised content or strict accuracy requirements, a custom pipeline gives you the control you need.
Conclusion
RAG is the most practical way to make a general-purpose AI model useful for your specific business. It lets the model answer questions about your products, your policies, your customers, and your operations — not from general knowledge, but from your actual documents and data.
But RAG is not a feature you add to a model. It is a system you build and maintain. The pipeline — ingestion, chunking, embedding, retrieval, and generation — has five stages, and a weakness in any stage propagates through to the output. The chunking strategy that splits a key passage across two chunks. The stale embedding that returns last quarter's policy as if it were current. The retrieval layer that misses a document because the user's terminology does not match the document's language. Each of these failures produces an answer that looks right and is not — which is worse than an answer that is obviously wrong, because the error is hard to catch.
The businesses that get RAG right are the ones that treat it as a continuous engineering discipline — building a solid pipeline, measuring retrieval quality, diagnosing failures, and improving the system every month. The businesses that treat RAG as a one-time integration get a system that slowly degrades and quietly produces wrong answers.
If you want a RAG system that works on your data — not just in a demo but in production, with real users asking real questions — we can help. We build the pipeline, tune the retrieval, and set up the evaluation discipline that keeps it honest.
Build a RAG system that works — we will ground your AI in your data, not guesswork.