A tiny RAG system over a folder of text with local embeddings

What you will build

Retrieval-augmented generation (RAG) is two programs pretending to be one. The first turns a pile of documents into vectors and finds the few pieces most similar to a question. The second writes an answer from those pieces. This project builds both at the smallest size that still behaves like the real thing, with no vector database, no framework and no hosted API. The retrieval half runs entirely on your CPU with a 22 M parameter embedding model. The generation half talks to a model running locally under Ollama.

The corpus is nine short notes on unrelated subjects (sourdough, bicycle gears, a lighthouse, compost, tides and so on) that the first block writes to a corpus/ folder. Unrelated subjects are the point. When the three test queries each pull chunks from exactly one file, you can see retrieval working without squinting. Swap the folder for your own notes afterwards and nothing else changes.

You should be comfortable with the embeddings lesson and the LLM lesson. The RAG pipeline checklist lists the decisions this project makes for you, so you know what to revisit when the corpus grows.

What you need

• Python 3.10 or later with numpy and sentence-transformers (pip install sentence-transformers, which pulls in torch and transformers). First use downloads the embedding model, about 90 MB. After that it runs offline from the local cache.
• httpx for the generation step (pip install httpx).
• For generation only: Ollama installed and running, with a model pulled. We used qwen2.5:7b (Qwen2.5 7B Instruct, 7.6 B parameters, 32 k context in the Ollama build, Apache-2.0, released September 2024), which is a 4.7 GB download in the default 4-bit quantisation and wants about 6 GB of free memory. Any instruction-tuned model Ollama offers works. Set OLLAMA_MODEL to use a different one.
• No GPU. The embedding model runs on CPU in well under a second for this corpus. Ollama uses the GPU on Apple Silicon automatically and falls back to CPU elsewhere.

The retrieval block was run from an empty directory with the Field Guide’s pipeline environment on an Apple Silicon Mac mini. The generation block was run on the same machine against a local Ollama. Timings are given with each block.

Design

Chunking. Each file is split into sentences, and chunks are windows of two consecutive sentences with a stride of one. A five-sentence file gives four chunks, and every sentence appears in two of them. The overlap means a fact split across a sentence boundary is still inside at least one chunk. For notes this short it is the right size. For longer documents you would chunk by paragraph or by a token budget of 200 to 500, and the checklist page covers the trade-off.

Embedding. sentence-transformers/all-MiniLM-L6-v2 (22.7 M parameters, 256 word-piece context, Apache-2.0, released 2021) maps a chunk to a 384-dimensional vector. With normalize_embeddings=True every vector has unit length, so the dot product of two of them is their cosine similarity. That turns the whole search into one matrix-vector product, index @ q, followed by argsort. For 36 chunks that is 14 thousand multiplications. It remains the right design up to a few hundred thousand chunks, which is further than most personal corpora get. Beyond that you add an approximate index such as FAISS or a vector database, and the code around it stays the same.

Why the same model for queries and chunks. The vector space only makes sense for one model. Embedding the query with a different model, or a different version of the same model, gives numbers that are not comparable with the index. Save the model name alongside the index.

Generation. The prompt numbers the retrieved chunks, names their files, and tells the model to answer only from them and to cite by number. Temperature 0 makes the output repeatable. The model’s answer is still the model’s answer: the retrieval guarantees it saw the right text, not that it reasoned correctly from it. One of the three test questions shows exactly that failure, and it is worth reading closely.

Code

Build the corpus, the index and the retriever

This block writes the nine files, chunks them, builds the index and answers three queries by printing the top three chunks with their scores and file names. It saves index.npy and chunks.tsv for the generation step.

# rag.py: write a small corpus, chunk it, embed it locally, retrieve by cosine similarity.
import re
import time
from pathlib import Path

import numpy as np
from sentence_transformers import SentenceTransformer

SEED = 0
np.random.seed(SEED)

# --- 1. A corpus of nine short notes on unrelated subjects ---
CORPUS = {
    "sourdough.txt": (
        "A sourdough starter is a culture of wild yeast and lactic acid bacteria kept alive in flour and water. "
        "Feed it once a day at room temperature or once a week in the fridge. "
        "A starter is ready to bake with when it doubles in volume within four to six hours of feeding. "
        "Bulk fermentation at 24 degrees Celsius takes about five hours for a dough with 20 percent starter. "
        "Bake in a preheated cast iron pot at 240 degrees Celsius, lid on for 20 minutes and lid off for 20 more."
    ),
    "bicycle-gears.txt": (
        "A bicycle drivetrain multiplies pedal cadence by the ratio of chainring teeth to sprocket teeth. "
        "A 50 tooth chainring with an 11 tooth sprocket gives a ratio of 4.5, which is a fast gear for descents. "
        "A 34 tooth chainring with a 34 tooth sprocket gives a ratio of 1.0 for steep climbs. "
        "Cross-chaining, running the big chainring with the big sprocket, bends the chain and wears it faster. "
        "A chain stretched beyond 0.75 percent of its nominal length should be replaced before it damages the cassette."
    ),
    "lighthouse.txt": (
        "The Eddystone Rocks lie 14 kilometres south of Plymouth and have carried four successive lighthouses. "
        "Smeaton's tower of 1759 was built from interlocking granite blocks shaped like tree trunks. "
        "It stood until 1877, when erosion of the rock beneath it forced a replacement. "
        "The current tower, designed by James Douglass, was lit in 1882 and automated in 1982. "
        "Smeaton's upper section was rebuilt on Plymouth Hoe, where it is open to visitors."
    ),
    "compost.txt": (
        "A compost heap needs a mix of nitrogen-rich green material and carbon-rich brown material. "
        "A ratio of roughly one part green to two parts brown by volume keeps the heap from going slimy. "
        "Turn the heap every two weeks to let air in, which keeps the aerobic bacteria working. "
        "A hot heap reaches 60 degrees Celsius in its core, which kills most weed seeds. "
        "Finished compost is dark, crumbly and smells of forest floor, usually after six to twelve months."
    ),
    "tides.txt": (
        "Tides are caused by the gravitational pull of the Moon and, to a lesser extent, the Sun. "
        "Spring tides, with the largest range, occur at new and full moon when the two pulls align. "
        "Neap tides, with the smallest range, occur at the quarter moons. "
        "The Bristol Channel has one of the largest tidal ranges in the world, up to 15 metres at Avonmouth. "
        "Tide tables predict high water to within a few minutes years in advance using harmonic analysis."
    ),
    "espresso.txt": (
        "A standard double espresso uses 18 grams of ground coffee to produce 36 grams of liquid in 25 to 30 seconds. "
        "Grind finer to slow the shot down and coarser to speed it up. "
        "Water should be between 92 and 96 degrees Celsius at the group head. "
        "A shot that runs too fast tastes sour and thin, while one that runs too slow tastes bitter and harsh. "
        "Clean the portafilter basket and backflush the machine weekly to keep old oils from spoiling the taste."
    ),
    "knots.txt": (
        "The bowline makes a fixed loop that will not slip and is easy to untie after loading. "
        "The clove hitch secures a rope to a post quickly but can work loose under a varying load. "
        "A figure of eight is the standard stopper knot and is also tied on a bight for climbing. "
        "The sheet bend joins two ropes of different diameters more reliably than a reef knot. "
        "A knot can reduce the breaking strength of a rope by 30 to 50 percent, so leave a margin."
    ),
    "honeybees.txt": (
        "A honeybee colony in summer holds one queen, up to 50,000 workers and a few hundred drones. "
        "Workers forage within about three kilometres of the hive and communicate flower locations with a waggle dance. "
        "The queen lays up to 2,000 eggs a day at the height of the season. "
        "A colony prepares to swarm when it runs out of space, raising a new queen in peanut-shaped cells. "
        "Beekeepers inspect hives every seven to ten days in spring to catch swarm preparations early."
    ),
    "woodstove.txt": (
        "Firewood should be seasoned to below 20 percent moisture before burning, which takes one to two years for oak. "
        "A moisture meter pressed into a freshly split face gives the most honest reading. "
        "A top-down fire, with kindling above the logs, lights cleanly with less smoke. "
        "Close the air control gradually once the fire is established to lengthen the burn. "
        "Have the chimney swept at least once a year to remove creosote, which is a fire risk."
    ),
}
corpus_dir = Path("corpus")
corpus_dir.mkdir(exist_ok=True)
for name, text in CORPUS.items():
    (corpus_dir / name).write_text(text)
print(f"wrote {len(CORPUS)} files to {corpus_dir}/")


# --- 2. Chunk each file into overlapping windows of sentences ---
def chunk_text(text: str, window: int = 2, stride: int = 1) -> list[str]:
    sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]
    chunks = []
    for start in range(0, max(1, len(sentences) - window + 1), stride):
        chunks.append(" ".join(sentences[start:start + window]))
    return chunks


chunks, sources = [], []
for path in sorted(corpus_dir.glob("*.txt")):
    for c in chunk_text(path.read_text()):
        chunks.append(c)
        sources.append(path.name)
print(f"{len(chunks)} chunks from {len(set(sources))} files")

# --- 3. Embed every chunk once and L2-normalise, so dot product = cosine ---
t0 = time.perf_counter()
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
index = model.encode(chunks, batch_size=32, normalize_embeddings=True, show_progress_bar=False)
index = np.asarray(index, dtype=np.float32)
print(f"index shape {index.shape}, built in {time.perf_counter() - t0:.1f} s")
np.save("index.npy", index)
Path("chunks.tsv").write_text("\n".join(f"{s}\t{c}" for s, c in zip(sources, chunks)))


# --- 4. Retrieve: embed the query, score every chunk, take the top k ---
def retrieve(query: str, k: int = 3) -> list[tuple[float, str, str]]:
    q = model.encode([query], normalize_embeddings=True)[0]
    scores = index @ q
    top = np.argsort(-scores)[:k]
    return [(float(scores[i]), sources[i], chunks[i]) for i in top]


QUERIES = [
    "How hot does the oven need to be for sourdough?",
    "When should I replace a worn bicycle chain?",
    "Why does the Bristol Channel have such big tides?",
]
for query in QUERIES:
    print(f"\nQ: {query}")
    for score, source, chunk in retrieve(query):
        print(f"  {score:.3f}  [{source}]  {chunk[:90]}...")

Expected output, 4.0 s on the Mac mini, of which 0.3 s is loading the model and embedding the 36 chunks:

wrote 9 files to corpus/
36 chunks from 9 files
index shape (36, 384), built in 0.3 s

Q: How hot does the oven need to be for sourdough?
  0.547  [sourdough.txt]  Bulk fermentation at 24 degrees Celsius takes about five hours for a dough with 20 percent...
  0.466  [sourdough.txt]  A starter is ready to bake with when it doubles in volume within four to six hours of feed...
  0.416  [sourdough.txt]  A sourdough starter is a culture of wild yeast and lactic acid bacteria kept alive in flou...

Q: When should I replace a worn bicycle chain?
  0.455  [bicycle-gears.txt]  Cross-chaining, running the big chainring with the big sprocket, bends the chain and wears...
  0.320  [bicycle-gears.txt]  A 34 tooth chainring with a 34 tooth sprocket gives a ratio of 1.0 for steep climbs. Cross...
  0.307  [bicycle-gears.txt]  A bicycle drivetrain multiplies pedal cadence by the ratio of chainring teeth to sprocket ...

Q: Why does the Bristol Channel have such big tides?
  0.749  [tides.txt]  The Bristol Channel has one of the largest tidal ranges in the world, up to 15 metres at A...
  0.677  [tides.txt]  Neap tides, with the smallest range, occur at the quarter moons. The Bristol Channel has o...
  0.510  [tides.txt]  Tides are caused by the gravitational pull of the Moon and, to a lesser extent, the Sun. S...

Every hit comes from the right file. Look at the scores, though. The top sourdough chunk scores 0.547 and the top chain chunk 0.455, and both of those chunks do contain the answer, but only in their second sentence, which the 90-character preview cuts off. Cosine scores from this model are not probabilities and rarely exceed 0.8 even for a perfect match. Do not threshold them at a round number without measuring on your own data. The overlap in the chunking is what put the 240 degrees sentence into the top chunk alongside the fermentation sentence that carried most of the similarity.

Generate an answer with citations

This block needs a running Ollama. It reloads the saved index, retrieves the top four chunks for each question, builds a numbered prompt and posts it to Ollama’s /api/generate route. It reads the server address and model name from the environment with sensible defaults.

# needs-api
# answer.py: retrieve with the saved index, then ask a local model through Ollama to answer with citations.
import os
from pathlib import Path

import httpx
import numpy as np
from sentence_transformers import SentenceTransformer

OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434")
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "qwen2.5:7b")
TOP_K = 4

# Load what rag.py saved: the normalised chunk embeddings and the chunk texts with their file names.
index = np.load("index.npy")
rows = [line.split("\t", 1) for line in Path("chunks.tsv").read_text().splitlines()]
sources, chunks = zip(*rows)
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")


def retrieve(query: str, k: int = TOP_K) -> list[tuple[str, str]]:
    q = embedder.encode([query], normalize_embeddings=True)[0]
    top = np.argsort(-(index @ q))[:k]
    return [(sources[i], chunks[i]) for i in top]


def build_prompt(question: str, hits: list[tuple[str, str]]) -> str:
    context = "\n\n".join(f"[{i + 1}] ({src})\n{text}" for i, (src, text) in enumerate(hits))
    return (
        "Answer the question using only the numbered sources below. "
        "Cite the sources you used as [1], [2] and so on after the sentence they support. "
        "If the sources do not contain the answer, say so.\n\n"
        f"Sources:\n{context}\n\nQuestion: {question}\nAnswer:"
    )


def ask_ollama(prompt: str) -> str:
    response = httpx.post(
        f"{OLLAMA_URL}/api/generate",
        json={"model": OLLAMA_MODEL, "prompt": prompt, "stream": False,
              "options": {"temperature": 0, "num_predict": 200}},
        timeout=120.0,
    )
    response.raise_for_status()
    return response.json()["response"].strip()


QUESTIONS = [
    "How hot does the oven need to be for sourdough?",
    "When should I replace a worn bicycle chain?",
    "Why does the Bristol Channel have such big tides?",
]
for question in QUESTIONS:
    hits = retrieve(question)
    answer = ask_ollama(build_prompt(question, hits))
    print(f"\nQ: {question}\nA: {answer}")
    print("sources: " + ", ".join(f"[{i + 1}] {src}" for i, (src, _) in enumerate(hits)))

Run

pip install numpy sentence-transformers httpx
python rag.py

Then, for the generation step, in another terminal:

ollama pull qwen2.5:7b
ollama serve

ollama serve is only needed if the Ollama desktop application is not already running. Back in the first terminal:

python answer.py

To use a different model, for example a smaller one on a machine with 8 GB of memory, set OLLAMA_MODEL=qwen2.5:3b before running.

Test

The output of rag.py is shown above and should match it line for line apart from the timing. For answer.py, this is what qwen2.5:7b produced on the Mac mini. The three questions took 22 s in total, almost all of it generation:

Q: How hot does the oven need to be for sourdough?
A: According to the sources provided, the oven needs to be preheated to 240 degrees Celsius for baking the sourdough. This information is cited from [1].
sources: [1] sourdough.txt, [2] sourdough.txt, [3] sourdough.txt, [4] sourdough.txt

Q: When should I replace a worn bicycle chain?
A: You should replace a worn bicycle chain when it stretches beyond 0.75 percent of its nominal length, as this can lead to damage of the cassette before it becomes fully worn out [1].
sources: [1] bicycle-gears.txt, [2] bicycle-gears.txt, [3] bicycle-gears.txt, [4] bicycle-gears.txt

Q: Why does the Bristol Channel have such big tides?
A: The Bristol Channel has such big tides because it experiences spring tides, which have the largest range, occurring at new and full moon when the gravitational pulls of the Moon and the Sun align [3, 4].
sources: [1] tides.txt, [2] tides.txt, [3] tides.txt, [4] tides.txt

The first two answers are correct, cite the right source and would satisfy anyone who asked. The third is the one to learn from. The corpus says the Bristol Channel has a large range. It never says why (the real reason is the funnel shape of the channel and resonance with the tidal period, which is not in any file). The instructions said to admit when the sources do not contain the answer. Instead the model took the nearest available mechanism, spring tides, and presented it as the explanation, with citations that look convincing. Retrieval did its job perfectly. Generation produced a confident wrong answer from correct sources, which is the field’s definition of a hallucination, and the citations did not prevent it. Every RAG system you build will do this sometimes. The checks that catch it are an evaluation set of questions whose answers are not in the corpus, and a reader who follows the citations.

A different model, or the same model on a different day’s Ollama build, may word all three answers differently. The citations and the shape of the failure are what to compare.

Things to try

• Point corpus_dir at a folder of your own notes or a project’s docs/ directory. Switch chunking to paragraphs (text.split("\n\n")) and print the chunk count and index build time. The index for a few thousand chunks still builds in under a minute on CPU.
• Add a question whose answer is not in the corpus (“What is the boiling point of mercury?”) and see whether the model admits it. Then tighten the prompt until it does, and check that the first two answers have not got worse.
• Print the score of the fourth hit for each query. When it comes from a different file than the top hit, it is adding noise to the prompt. Try TOP_K = 2 and a score cut-off relative to the top score (say, 80 % of it) instead of a fixed k.
• Replace cosine retrieval with BM25 keyword scoring (the rank_bm25 package) and compare the top three for the chain question. Then combine the two by rank. Hybrid retrieval is the usual next step and the checklist page explains why.
• Save the model name into chunks.tsv as a header line and refuse to run answer.py if it differs from the embedder loaded. It prevents a silent mismatch when you upgrade the embedding model.
• Replace the Ollama call with a hosted API, keeping the prompt identical, and compare the three answers. The retrieval half of the system does not change at all.

Summary

• A RAG system is a retriever and a generator. Build and test them separately.
• Overlapping sentence windows, a sentence-transformers model and one numpy matrix product make a complete retriever for corpora up to the hundreds of thousands of chunks.
• Normalise embeddings once so cosine similarity is a dot product, and always embed queries with the same model as the index.
• Number the retrieved chunks in the prompt and ask for citations by number, then check the citations yourself.
• Correct retrieval does not guarantee a correct answer. Keep questions with no answer in the corpus in your test set.