What is RAG? Retrieval-Augmented Generation Explained

RAG (retrieval-augmented generation) is a technique that lets an AI model look up relevant information from your own documents before answering, so its responses are grounded in your actual business data rather than guesswork.

What is RAG

RAG stands for retrieval-augmented generation. It is a way of connecting an AI language model to your own information - your quotes, manuals, policies, past jobs - so the AI answers from those documents instead of relying only on what it learned during training.

Think of it as giving the AI an open-book exam. Without RAG, a model answers from memory, which is where hallucinations come from. With RAG, the system first retrieves the most relevant passages from your knowledge base, then generates an answer using those passages as evidence. Good RAG systems also cite which document the answer came from, so you can check it.

How RAG works

A RAG system has two halves: retrieval and generation. Here is the flow when someone asks a question.

Your documents are split into chunks and converted into embeddings - numerical representations of meaning - which are stored in a vector database.
When a question comes in, it is converted the same way, and the database finds the chunks whose meaning is closest to the question. This works even when the wording is completely different.
Those chunks are handed to the language model along with the question, with an instruction along the lines of "answer using only this information".
The model writes an answer grounded in the retrieved material, ideally citing its sources.

Because the knowledge lives in a database rather than inside the model, updating the system is as simple as updating the documents. No retraining, no fine-tuning.

What RAG means for NZ small businesses

RAG is the most practical way for a small business to get an AI assistant that actually knows the business. You do not need a custom model or a big budget - you need your existing documents and a well-built retrieval layer over them.

Some concrete examples. A trades business could put its quoting history and supplier price lists behind RAG, so an estimator can ask "what did we charge for a reroof on a 1990s townhouse" and get an answer drawn from real past jobs. An engineering or building firm could make company standards and compliance documents searchable in plain English, with answers that cite the exact clause. An accounting or legal practice could let staff query years of internal precedents and templates instead of digging through shared drives.

The common thread: the knowledge already exists in your business, it is just slow to find. RAG turns that buried knowledge into answers in seconds, and it keeps client data inside your own systems rather than inside a model.

Getting started

Start small and prove value on one document set before expanding. The quality of a RAG system depends far more on the quality and organisation of your documents than on the AI model behind it.

Pick one high-value knowledge base - quoting history, SOPs, or your compliance folder - where staff currently waste time searching.
Tidy it first. Remove outdated versions and duplicates, because RAG will faithfully retrieve wrong information too.
Trial an off-the-shelf option before building custom - Claude Projects or a similar tool can act as a lightweight RAG layer over uploaded documents.
Test it with real questions from your team and check the citations before rolling it out wider.

Frequently asked questions

Is RAG better than fine-tuning a model?

Does RAG stop AI hallucinations?

Is my business data safe in a RAG system?

Want help putting this to work in your business?

Book a free discovery call and we’ll map where RAG (Retrieval-Augmented Generation) fits your workflows - in plain English, grounded in your actual systems.

Book a Free Discovery Call

Last updated 13 July 2026