Why Your RAG Chatbot Gives Bad Answers (and How to Fix It, Step by Step)

The short answer: Most bad RAG answers are retrieval failures, not generation failures: the passage that contains the answer never reached the model. To fix it, build a small set of real questions with known answers, measure retrieval on its own (recall@k), then work through the usual culprits in order: chunking that splits answers apart, pure vector search missing exact terms, vague queries, too few candidates without reranking, and stale or duplicate documents. Tune the prompt last.

Every retrieval-augmented generation project seems to follow the same arc.

Week one is magic. You load your docs into a vector database, wire up an embedding model, write a twenty-line prompt, and ask it ten questions. It answers nine of them perfectly. Someone records a demo. Leadership is thrilled.

Week six is a support channel full of screenshots. The bot says the refund window is 14 days when the policy page clearly says 30. It can't find the setup steps for the enterprise SSO integration, even though there's a whole page about it. When asked about error code E-4012, it confidently explains a different error. And the fun one: someone asks "what about for annual plans?" as a follow-up and gets an answer about something else entirely.

The usual reaction is to start changing things at random. Swap the embedding model. Try a different vector database. Rewrite the prompt in all caps. Add "you are a world-class expert" to the top. Sometimes one of these helps a little, and nobody knows why, and three weeks later it gets worse again.

This guide is the alternative: a methodical way to find out where your RAG pipeline is failing and fix the actual cause. It's written for engineers who already have something working and need it to work reliably. The examples use Python, but the ideas apply to any stack.

The two-stage mental model

A RAG system does two separate jobs, and they fail in completely different ways.

1. Retrieval: given the user's question, find the handful of passages from your corpus that contain the answer.
2. Generation: given the question and those passages, write a correct, grounded answer.

When a user reports a bad answer, you can't tell from the answer alone which stage broke. But you can tell in about thirty seconds by looking at what was retrieved. For any failed question, ask three things in order:

If you do this for twenty or thirty real failures, you'll almost certainly find that retrieval accounts for most of them. That's the main insight of this whole article, and it's why tweaking the prompt so rarely fixes anything: the model can't quote a paragraph it was never shown.

To make this check easy, log everything. For every request, store the original question, any rewritten query, the IDs and scores of every retrieved chunk, the final prompt sent to the model, and the response. Without this, debugging RAG is guesswork. With it, most problems become obvious within minutes.

Step zero: build a tiny evaluation set

Before changing anything, you need a way to tell whether a change helped. "It seems better" is how teams end up going in circles.

You don't need a huge benchmark. Thirty to fifty questions is enough to start, as long as they're real. Pull them from your logs, your support tickets, and the questions that triggered complaints. For each one, record which document (or ideally which section) contains the answer. A spreadsheet is fine. Make sure the set includes the hard stuff: questions with exact identifiers, follow-up questions, questions that need two documents, and a few questions your corpus genuinely can't answer.

Then measure retrieval separately from generation. Two metrics cover most needs:

• Recall@k: for what fraction of questions did at least one correct chunk appear in the top k results? If recall@5 is 60%, then four in ten questions never had a chance.
• MRR (mean reciprocal rank): how high did the first correct chunk rank? Finding it at position 1 scores 1.0, at position 4 scores 0.25. This tells you whether a reranker would help.

from dataclasses import dataclass


@dataclass
class EvalCase:
    question: str
    relevant_doc_ids: set[str]  # any of these counts as a hit


def recall_at_k(cases: list[EvalCase], retrieve, k: int = 5) -> float:
    hits = 0
    for case in cases:
        retrieved = [chunk.doc_id for chunk in retrieve(case.question, k)]
        if case.relevant_doc_ids & set(retrieved):
            hits += 1
    return hits / len(cases)


def mean_reciprocal_rank(cases: list[EvalCase], retrieve, k: int = 20) -> float:
    total = 0.0
    for case in cases:
        retrieved = [chunk.doc_id for chunk in retrieve(case.question, k)]
        for rank, doc_id in enumerate(retrieved, start=1):
            if doc_id in case.relevant_doc_ids:
                total += 1 / rank
                break
    return total / len(cases)

Run this before and after every change. It takes seconds, costs nothing beyond embedding the questions, and turns arguments into numbers. When you later want to evaluate the generated answers too, the approach in how to test LLM features with evals builds directly on this set.

Failure 1: chunking cut the answer in half

Chunking is the least glamorous part of RAG and the most common cause of retrieval failures. Most tutorials split documents into fixed-size pieces, say 500 tokens with a bit of overlap, without looking at the structure. That causes several predictable problems.

• Orphaned content. A section heading lands at the end of one chunk and its content at the start of the next. The chunk with the actual answer no longer mentions what it's about.
• Broken tables and lists. A pricing table gets split mid-row. The chunk with the price no longer contains the plan name.
• Lost context. A chunk says "this setting must be enabled before migrating" and never names the product, because that was three paragraphs up.

That last one is subtle and very common. Imagine a chunk that reads: "Refunds are available within 30 days of the renewal date." Which plan? Which product? The paragraph above said "For annual plans", but that line went into a different chunk. Now a question about annual-plan refunds has to match a chunk that never says "annual".

Fix: chunk by structure and carry the context with you

Split along the document's natural boundaries (headings, sections, list items) and prepend each chunk with its breadcrumb trail: document title plus the heading path. The text you embed then carries its own context.

import re


def chunk_markdown(doc_title: str, markdown: str, max_chars: int = 2000):
    """Split on headings and prefix each chunk with its heading path."""
    chunks, path, buffer = [], [], []

    def flush():
        body = "\n".join(buffer).strip()
        if body:
            breadcrumb = " > ".join([doc_title, *path])
            # Very long sections still get split, but every piece keeps its breadcrumb.
            for start in range(0, len(body), max_chars):
                chunks.append(f"{breadcrumb}\n\n{body[start:start + max_chars]}")
        buffer.clear()

    for line in markdown.splitlines():
        heading = re.match(r"^(#{1,6})\s+(.*)", line)
        if heading:
            flush()
            level = len(heading.group(1))
            path[:] = path[: level - 1] + [heading.group(2).strip()]
        else:
            buffer.append(line)
    flush()
    return chunks

This is deliberately simple. In production you'd split long sections on paragraph boundaries rather than raw character counts and handle tables as whole units, but even this version fixes the orphaned-heading problem.

Anthropic published a more powerful version of the same idea in 2024, called contextual retrieval. Before indexing, an LLM writes a short sentence or two situating each chunk within its full document, and that context is prepended to the chunk for both embedding and keyword indexing. In their experiments, contextual embeddings combined with contextual BM25 reduced the top-20 retrieval failure rate by 49%, and adding reranking brought the reduction to 67%. It costs one LLM call per chunk at indexing time, which prompt caching makes much cheaper, and nothing extra at query time.

Choosing a chunk size

There's no universal number, and anyone who gives you one without seeing your documents is guessing. Small chunks are precise but lose context. Large chunks keep context but dilute the embedding, so a chunk that covers five topics matches none of them strongly. A reasonable starting range for prose documentation is a few hundred to roughly a thousand tokens, aligned to sections. Then test two or three sizes against your eval set and let recall@k decide.

Failure 2: vector search can't find exact terms

Embeddings capture meaning, which is wonderful when someone asks "how do I get my money back" and the doc says "refund policy". They're much worse at exact tokens that carry no semantic meaning on their own: error codes, product SKUs, version numbers, function names, ticket IDs, people's names, and internal acronyms.

Ask a pure vector search about E-4012 and it will happily return passages about E-4021 and E-4102, because in embedding space they all look like "an error code from this product". For technical documentation and support content, where users paste identifiers all the time, this is a constant source of wrong answers.

Fix: hybrid search with reciprocal rank fusion

Keyword search, typically BM25, has the opposite strengths: it nails exact matches and fails at paraphrase. Run both, then merge the results. The standard way to merge two ranked lists without worrying about their incomparable scores is reciprocal rank fusion (RRF), described by Cormack, Clarke and Buettcher in 2009. Each document scores 1 / (k + rank) in each list, and the scores are summed. The constant k is conventionally 60.

from collections import defaultdict


def reciprocal_rank_fusion(*ranked_lists: list[str], k: int = 60) -> list[str]:
    """Merge several ranked lists of chunk IDs into one."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in ranked_lists:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] += 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)


def hybrid_search(query: str, top_n: int = 50) -> list[str]:
    semantic = vector_index.search(embed(query), limit=top_n)
    lexical = bm25_index.search(query, limit=top_n)
    return reciprocal_rank_fusion(semantic, lexical)[:top_n]

Many databases now support this natively: Postgres can combine pgvector with full-text search, and Elasticsearch, OpenSearch, Weaviate, Qdrant and others offer hybrid modes. Use the built-in version if you have it. The point is to have both signals.

If your eval set has identifier-heavy questions, adding BM25 is often the single biggest jump you'll see in recall. It's also cheap, boring, and well understood, which are three of my favorite properties in production software.

Failure 3: the question doesn't look like the answer

Users write short, vague, conversational queries. Documentation is written in long, formal, declarative sentences. "sso not working" and "Configuring SAML-based single sign-on for Enterprise workspaces" are about the same thing, but their embeddings may not be as close as you'd hope.

Conversation makes it worse. In a multi-turn chat, the user's latest message often makes no sense alone. "What about annual plans?" contains almost no retrievable information. If you embed that message directly, you'll retrieve whatever happens to mention annual plans most prominently, which might be completely unrelated to the refund question that came before.

Fix: rewrite the query before retrieving

Several techniques help, and you can combine them:

• Standalone question rewriting. For multi-turn chats, use a fast, cheap model call to rewrite the latest message into a complete question using the conversation history: "What is the refund policy for annual plans?" This one fix eliminates a whole category of follow-up failures and should be standard in any chat-style RAG app.
• Multi-query expansion. Generate three or four phrasings of the question, retrieve for each, and fuse the results with RRF. It improves recall for vague questions at the cost of a few more searches.
• HyDE (hypothetical document embeddings). Ask a model to write a plausible answer to the question, then embed that and search with it. The fake answer is phrased like documentation, so it lands closer to the real documentation in embedding space. The technique comes from Gao et al., 2022. The hypothetical answer is only used for searching, never shown to the user.

Each of these adds latency and cost, so measure before adopting. Standalone rewriting for chat is almost always worth it. The others depend on your data.

Failure 4: the right chunk was found, then thrown away

Here's a pattern that shows up constantly once you start logging ranks: the correct chunk was retrieved, but at position 9 or 14, and the pipeline only sends the top 5 to the model. Your recall@20 looks healthy while your recall@5 is poor. MRR tells the same story.

First-stage retrieval, whether vector or hybrid, is built to be fast over millions of chunks. It compresses each chunk into a single vector or keyword statistics before it ever sees the question. That's great for narrowing a haystack, less great for precise ordering.

Fix: retrieve wide, rerank narrow

Retrieve a larger candidate set, say 30 to 100 chunks, and then reorder them with a reranker. Rerankers are usually cross-encoders: models that read the question and a candidate chunk together and output a relevance score. Because they see both texts at once, they're far more accurate than comparing two precomputed vectors. They're also much slower, which is why you only run them on a short candidate list. Hosted reranking APIs and open-source cross-encoders are both widely available.

After reranking, pass the top handful to the model. In Anthropic's contextual retrieval experiments, reranking was the step that took the failure reduction from 49% to 67%, which matches what you'll typically see: rerankers are one of the highest-return additions to a pipeline that already has decent recall at larger k.

Failure 5: the corpus disagrees with itself

Some wrong answers are not retrieval bugs or model bugs. The system retrieved a passage that genuinely says the wrong thing, because your corpus contains the 2023 version of the policy, the 2025 version, a Slack export where someone guessed, and a draft that was never published. The model picks one, or worse, blends them into a confident answer that matches none.

Duplicate and near-duplicate content causes a related problem: five copies of the same boilerplate paragraph crowd the top results and push out the one chunk that was actually specific to the question.

Fix: curate and add metadata

• Remove or archive superseded documents from the index. An index is not an archive.
• Deduplicate chunks with an exact hash, and consider near-duplicate detection for boilerplate like footers and legal text.
• Store metadata with every chunk (source, product, version, last updated date, audience) and filter on it. If the user is on the enterprise plan, don't retrieve consumer-plan docs.
• Include the date and source in the context you pass to the model, so it can prefer newer material and the user can verify it.

This work is unglamorous and nobody gives conference talks about deleting stale PDFs. It's also one of the most effective quality improvements you can make.

Failure 6: the model had the answer and still got it wrong

Once retrieval is in good shape, the remaining failures are usually in generation. A few patterns account for most of them.

Too much context. More chunks feel safer, but models don't use long contexts evenly. The paper "Lost in the Middle" (Liu et al., 2023) found that models were best at using information near the beginning or end of their context and noticeably worse at using information buried in the middle. Newer models have improved, but the practical lesson holds: five highly relevant chunks usually beat twenty mixed ones. Put the strongest material first.

Prior knowledge overriding your docs. If your product does something unusual, the model may "correct" your documentation toward what's typical. Tell it explicitly to answer only from the provided sources, and to say so when the sources don't contain the answer.

No permission to say "I don't know". If your prompt implies that an answer is always expected, the model will produce one. Give it an explicit, acceptable way out, and make sure your interface handles that case gracefully with a link to search or a human.

Fix: ask for citations and check them in code

Give every chunk an ID in the prompt and require the model to cite the IDs it used. Then verify in code that every cited ID was actually in the context, and consider checking that quoted text appears in the cited chunk. Citations don't make the model correct, but they make errors visible to users and much easier to debug for you.

import re

SYSTEM_PROMPT = """Answer the question using only the sources below.
Cite sources inline like [S2]. If the sources do not contain the answer,
say you could not find it in the documentation. Do not guess."""


def build_context(chunks) -> str:
    return "\n\n".join(
        f"[S{i}] (source: {c.source}, updated: {c.updated_at:%Y-%m-%d})\n{c.text}"
        for i, c in enumerate(chunks, start=1)
    )


def invalid_citations(answer: str, num_chunks: int) -> list[str]:
    cited = set(re.findall(r"\[S(\d+)\]", answer))
    return [c for c in cited if not 1 <= int(c) <= num_chunks]

If invalid_citations returns anything, the model invented a source. Log it, and either retry or show the answer with a warning. An answer with no citations at all, when the question clearly needed one, is another useful signal to track.

Failure 7: it answered with something the user shouldn't see

This one isn't about quality, but it's the failure that gets people fired, so it belongs on the list. If your index mixes documents with different access levels (HR policies, customer contracts, internal incident reports), then a user can retrieve content they aren't allowed to read just by asking the right question.

The fix is to enforce permissions at retrieval time, as a filter in the search query, based on the authenticated user. Never retrieve everything and ask the model to "only use documents the user has access to". The model is not an access-control system, and a determined user will talk it out of that instruction. Retrieved content is also a vector for prompt injection, so anything users can write into your index should be treated as untrusted text.

The debugging runbook

When RAG quality is poor and you don't know where to start, work through this list in order. Each step is cheaper than the next, and earlier steps often make later ones unnecessary.

1. Log everything: question, rewritten query, retrieved chunk IDs and ranks, final prompt, answer.
2. Build the eval set: 30 to 50 real questions, each mapped to the documents that answer it.
3. Triage failures into content gaps, retrieval failures, and generation failures.
4. Fix content gaps by writing or updating documentation. No code required.
5. Fix chunking: structure-aware splits, heading breadcrumbs, consider contextual retrieval.
6. Add keyword search and fuse with vector results using RRF.
7. Rewrite queries, at minimum turning follow-ups into standalone questions.
8. Retrieve wider and rerank, then pass fewer, better chunks to the model.
9. Curate the corpus: remove stale docs, deduplicate, add metadata filters.
10. Tune generation: source-only instructions, an explicit "not found" path, verified citations.

Notice what's missing from the first several steps: changing your embedding model, switching vector databases, fine-tuning, and building an agent. These can matter eventually, but they're expensive and rarely the cause. A better embedding model won't reattach a heading that your chunker cut off.

Signs of a healthy RAG system

It's worth setting expectations, because RAG will never be perfect and chasing 100% leads to over-engineering. A healthy production system usually looks like this:

• Retrieval recall at your chosen k is high on your eval set, and you know which question types still fail.
• When the answer isn't in the corpus, the bot says so instead of improvising.
• Answers cite sources, and users can click through to verify.
• Every quality complaint can be traced to a logged request and triaged in minutes.
• Failures turn into new eval cases, so the same bug doesn't come back silently.

That last point is what separates teams who improve steadily from teams stuck in the demo-to-complaints loop. The eval set is not a one-time project. It's a living record of everything that has gone wrong, and your guarantee that it won't go wrong the same way twice.

Frequently asked questions

Why does my RAG chatbot give wrong answers?

In most systems the main reason is that retrieval didn't return the passage containing the answer, often because chunking split it from its context, vector search missed an exact term, or the right chunk ranked too low to be included. Check the retrieved chunks for failed questions before changing the prompt or model.

What is a good chunk size for RAG?

There's no universal best size. For prose documentation, a few hundred to about a thousand tokens per chunk, split along headings and sections, is a sensible starting point. Test two or three sizes against a set of real questions and choose based on retrieval recall rather than intuition.

Do I need a vector database for RAG?

Not necessarily. For small or medium corpora, Postgres with the pgvector extension, or even keyword search alone, can work well. What matters more is good chunking, combining semantic and keyword search, and reranking. Choose a dedicated vector database when scale or features require it, not by default.

Is long context replacing RAG?

For small, stable document sets, putting everything into a long context window can be simpler and work well. For large, changing, or permission-controlled corpora, retrieval is still more practical because of cost, latency, and access control, and because models don't use very long contexts evenly. Many systems combine both: retrieve generously, then use a long context.

How do I measure RAG quality?

Measure retrieval and generation separately. For retrieval, use recall@k and mean reciprocal rank on a set of real questions mapped to the documents that answer them. For generation, check answer correctness and whether claims are supported by the cited sources, using a mix of code checks, a calibrated LLM judge, and human review.

The takeaway

RAG systems look like AI problems, but most of their failures are search problems wearing an AI costume. The model at the end of the pipeline is usually doing its best with what you gave it. Give it the right paragraphs, in a sensible order, with enough context to understand them, and it will answer correctly most of the time.

So resist the urge to swap models. Open the logs, look at what was retrieved, and fix the boring stuff first. It's less exciting than a new architecture diagram, and it works.