What is retrieval augmented generation?

Chunking. Documents are cut into passages small enough to be retrieved precisely and large enough to still make sense alone. This sounds mechanical and is not: where the cuts fall determines what can be found later, and a passage severed from the context that gave it meaning is worse than useless because it is still retrievable.

Embedding. Each passage is converted into a vector, a list of numbers positioning it in a space where things that mean similar things sit near each other. This is what lets the system find a passage about a late payment when somebody asks about an overdue invoice, without the two sharing a single word.

Search. The question is embedded the same way, and the system retrieves the passages nearest to it. In practice pure vector search is usually combined with ordinary keyword search, because semantic similarity is poor at exactly the things keywords are good at: names, reference numbers, and precise terms of art.

Reranking. The first pass is tuned for recall, so it returns more than is needed. A second, more expensive pass reorders those candidates by how well each actually answers the question and discards the rest. Skipping this step is one of the more common reasons a system demos adequately and disappoints in use.

Generation, constrained. The surviving passages go to the model along with the question, and the model is instructed to answer from them rather than from its own knowledge, and to say so when they do not contain an answer. That constraint is the whole point. An unconstrained model asked the same question will produce something plausible regardless of whether the material supports it.

A retrieval answer without a citation is a rumour with good grammar. The passages are right there in the pipeline, so returning a pointer to the source alongside the answer costs almost nothing, and refusing to is a decision rather than a limitation.

What citations buy is that the system can be wrong safely. Somebody reading an answer can open the source and see immediately whether it supports what was said, which turns a wrong answer into a visible error rather than a silent one. For any work that carries consequences, that property matters more than raw accuracy does.

HDS has shipped this. In a retrieval layer built over a professional firm's correspondence, case-relevant correspondence resolves by search rather than by manual review. Results return with thread citations attorneys can open and verify.

It also changes how the system gets adopted. People extend trust to something they have personally spot-checked a few times, and citations are what make spot-checking possible at all. Without them the only options are believing it or ignoring it, and professionals sensibly choose the second.

Retrieval is also the tool an agent reaches for most often, so if the surrounding question is what an autonomous system does with these answers, that is covered in what an AI agent is.

Chunking that splits meaning. A passage cut in the wrong place keeps the words and loses the context. A reply severed from the message it answers, or a clause separated from the definition it depends on, will still be retrieved and will still read as authoritative. Chunk boundaries have to follow the structure of the material rather than a character count, which is why chunking a mail corpus and chunking a contract are different jobs.

Confidently retrieving the wrong document. Vector search always returns its nearest neighbours, and when nothing relevant exists it returns the nearest irrelevant thing instead, with no signal that it has done so. A system that cannot say it does not know will confabulate from whatever it happened to fetch. Designing against this means relevance thresholds, reranking, and an explicit path for answering that the material does not cover the question.

Access control that is too coarse. Permission usually has to hold per matter, per client, or per team rather than across the corpus as a whole, and it has to be enforced when passages are fetched rather than applied to the answer afterwards. A system that retrieves everything and then filters has already read what it should not have, and one bad prompt away from repeating it.

Retention is the first decision, and it is a policy question rather than a technical one. What material enters the index, how long it stays, and what happens when the underlying document is deleted all have to be settled deliberately, because an index quietly outliving the documents it was built from is a records problem nobody intended to create.

Access logging is the second. Being able to reconstruct who asked what, and which documents were surfaced in reply, is what makes a retrieval system reviewable later by somebody who was not there. In regulated and professional-services settings that is frequently a requirement rather than a nicety.

Both were part of the shipped deployment rather than added afterwards. The engagement covered a secure Microsoft 365 pipeline with controlled retention and audit-ready access logging, built alongside the retrieval layer because retrofitting either onto a system already holding confidential material is considerably harder than designing them in.