JymboAI

Learn

What is RAG? Teach AI to use your own documents

AI models only know what they were trained on. Retrieval-augmented generation (RAG) lets them answer from your own files. Here is how it works, when to use it and how to start.

Beginner3 min readBy JymboAI

Ask a general AI assistant about your company's refund policy, your project notes or last week's meeting, and it can't know the answer. It was never trained on your documents. Retrieval-augmented generation, usually called RAG, fixes that.

The short version

RAG gives an AI model an open-book exam. Before it answers, the system retrieves the most relevant passages from your documents and hands them to the model along with your question. The model then generates an answer based on those passages.

Key idea: The model doesn't memorize your documents. It looks things up each time, like a person checking the manual before answering.

How it works, step by step

Getting your documents ready (done once, then updated):

  1. Collect: gather the files: PDFs, Word documents, web pages, notes.
  2. Convert: turn them into clean text. Tables and headings matter, and messy conversion is a common cause of bad answers.
  3. Chunk: split the text into small passages, often a few paragraphs each.
  4. Embed: convert each chunk into an embedding, a list of numbers that captures its meaning, so similar ideas end up close together.
  5. Store: save the embeddings in a vector database so they can be searched quickly.

Answering a question (every time):

  1. Your question is turned into an embedding too.
  2. The database finds the chunks closest in meaning to it.
  3. Those chunks and your question go to the AI model.
  4. The model answers using the chunks, ideally citing which document each fact came from.
Question ──► find matching chunks ──► question + chunks ──► AI model ──► answer + sources

Why not just retrain the model?

Retraining or fine-tuning a model on your documents is possible, but for most "answer from my files" needs, RAG is the better first choice:

RAGFine-tuning
Add or update a documentRe-index it in minutesRetrain the model
Show where an answer came fromYes, it can cite sourcesNot directly
Cost to startLowHigher
Best forFacts and knowledge that changeTeaching a style, format or specialized behavior

The two can also be combined, but start with RAG.

Where RAG goes wrong

  • Bad input: scanned pages, broken tables or jumbled text lead to wrong answers. Clean conversion tools help a lot.
  • Wrong chunks retrieved: if the search finds the wrong passages, the answer will be wrong too, however good the model is.
  • Chunks too big or too small: too small loses context, too large dilutes the match. It's worth experimenting.
  • Confident gaps: if the answer isn't in your documents, a model may still guess. Instruct it to say "I don't know" when the sources don't cover a question.
  • Stale knowledge: re-index documents when they change.

Warning: RAG doesn't make data private by itself. If documents or questions are sent to a cloud model, they leave your computer. For sensitive files, use a local model, or check exactly where your data goes.

How to start

No code: try a desktop app such as AnythingLLM (opens in a new tab). Add a few of your own documents and ask questions only they can answer.

With code: a common first stack is a framework such as LlamaIndex (opens in a new tab) with an embedded vector database such as Chroma (opens in a new tab).

Messy documents: run them through a converter such as Docling (opens in a new tab) first, and compare the results.

Each of these tools is listed with more detail on our Resources page.

A simple test for any RAG setup

Write ten questions whose answers you know are in your documents, plus two whose answers are not. A good setup answers the ten correctly with sources, and admits it doesn't know the other two. Re-run the same questions whenever you change something.

Where to go next