How RAG Picks Which Passages to Pull From Your Site

When an AI answer engine responds to a question with a cited source, most of the work happened before the model wrote a single word. A retrieval step decided which passages were relevant, ranked them, and handed the top few to the model as context. The model mostly summarizes what it was given. If your page never made it into that context window, you were never in the running, no matter how good the writing was.

This is retrieval-augmented generation, or RAG. Understanding how the retrieval half works tells you more about getting cited than any amount of prompt speculation. Below is the actual mechanism, in plain terms, and what it means for how you structure a page.

What retrieval-augmented generation actually does

RAG is a two-stage process: retrieve, then generate. Instead of relying only on what a language model memorized during training, the system first searches a knowledge source — a web index, a vector database, a set of documents — for passages relevant to the query. It pulls the best-matching chunks, inserts them into the prompt alongside the user's question, and asks the model to answer using that material.

The important word is chunks. RAG systems do not store or retrieve whole articles. During indexing, your page gets split into smaller segments, often a few hundred words each, sometimes a single paragraph or a couple of sentences. Each chunk is converted into a vector — a long list of numbers representing its meaning — and stored. When a question comes in, the question is also converted to a vector, and the system finds the chunks whose vectors sit closest to it. That closeness is semantic similarity, not keyword matching.

So the unit of competition is not your article. It is the individual passage. A 2,000-word guide might contribute exactly one 120-word chunk to an answer, and the rest of the page is irrelevant to that query.

How does RAG decide which passages are relevant?

RAG ranks passages by semantic closeness to the question, then usually re-ranks the top candidates with a second, more precise model before feeding a handful into the prompt. The first pass is fast and approximate: it compares the question vector against millions of chunk vectors and returns maybe the top 50 or 100. Because vectors capture meaning, a chunk that says "most systems fail because the condensate line clogs" can match a question about "why is water leaking from my AC" even with no shared keywords.

The second pass is a re-ranker. It takes the question and each candidate chunk together and scores how well the chunk actually answers that specific question, not just how topically similar it is. This step filters out chunks that are on-topic but off-target. After re-ranking, the system keeps the top few — often three to eight — and those become the context window. Everything below the cut does not exist as far as the answer is concerned.

Three factors decide whether your chunk survives:

• Semantic match — does the passage genuinely address the meaning of the question, not just mention the subject.
• Self-containment — can the chunk be understood on its own, without the paragraph before it or a heading three sections up.
• Answer density — does it state a claim or answer directly, rather than winding up to one.

Why chunking is where most pages lose

Here is the failure that plays out constantly. A page has the right answer, but the answer is spread across three paragraphs with the key fact in the middle one, and the surrounding sentences depend on context established earlier under a different heading. When that middle paragraph gets isolated as a chunk, it reads like a fragment. It references "this method" or "the second option" without saying what those are. The re-ranker sees a chunk that gestures at an answer without delivering one, and scores it low.

Compare two ways of writing the same information.

Weak for retrieval

"As mentioned above, this is usually the culprit. Left unchecked, it leads to the problems we discussed, which is why the earlier step matters so much." Pulled out of the page, this chunk answers nothing. It is a connective passage, not a standalone one.

Strong for retrieval

"A clogged condensate drain line is the most common cause of water leaking from a central AC unit. The line backs up, the drain pan overflows, and water ends up on the floor near the indoor air handler." This chunk names the subject, states the answer, and explains the mechanism in three sentences. It survives isolation.

The second version does not require you to write worse or more repetitively across a whole page. It requires that each section be able to stand up if someone quoted only that section.

A worked example: making a page RAG-ready

Suppose a commercial roofing company in Kansas City has a page targeting "how long does a TPO roof last." Here is a sequence to make it retrievable.

1. Put the direct answer in its own paragraph, high up. "A commercial TPO roof lasts 15 to 20 years with proper installation and regular inspection." One sentence, self-contained, no pronoun that points elsewhere.
2. Give each sub-question its own heading and its own answerable chunk. "What shortens TPO roof lifespan?" followed by a paragraph that opens with the answer, then lists causes. The heading helps indexing systems segment cleanly on a natural boundary.
3. Repeat the subject noun instead of using "it." Within a chunk, write "a TPO membrane" rather than "it" the third time, so the chunk names its subject even when detached from the heading.
4. Keep factual chunks tight. Aim for the core answer inside the first two sentences of any paragraph meant to be pulled. Supporting detail after that is fine; just do not bury the claim.
5. Use a table for structured comparisons. Lifespan by membrane type, for instance, gives retrieval a compact, high-density block that answers comparison questions cleanly.

Notice the table cells are self-describing. "15–20 years" sitting next to "TPO" in a labeled column carries its own meaning; a retrieval system can lift the row and it still makes sense.

Does formatting content this way hurt readers?

No — writing self-contained sections improves the page for human readers as much as for retrieval systems, because people skim and jump the same way machines chunk. Someone landing on your roofing page from a search rarely reads top to bottom. They scan headings, stop at the one matching their question, and read that block. If that block only makes sense after reading the previous three, you have already lost them. The discipline that makes a passage retrievable — clear heading, answer first, subject named, claim before elaboration — is the same discipline that makes a page usable when someone is standing in a mechanical room on their phone.

The one place to be careful is repetition. Naming the subject in every chunk can feel redundant if you read the whole page straight through. That is a real tradeoff, but a mild one, and you resolve it by varying phrasing rather than by leaning on pronouns that break when a chunk is isolated.

What this changes about how you plan content

Plan at the level of the answerable question, not just the article. Before writing, list the specific questions a section should be able to answer on its own, and make sure each gets a heading and a self-contained response. Audit existing pages by copying a single paragraph out of context and reading it cold — if you cannot tell what it is about, retrieval systems can't either. This is the core of how we structure client pages at ClearPath Content: map the question-space first, then write each section so it survives being quoted alone.

The practical takeaway: RAG competes at the passage level, so write passages that stand on their own. Answer first, name the subject, keep factual chunks tight, and give every distinct question its own heading. A page built that way gets pulled into more answers and reads better for the person who actually landed on it.