Skip to content

Lesson 3 of 6

Retrieve the Most Relevant Chunks

Rank chunks by cosine similarity to a question and select the top few to provide context.

7 minAI Engineering

After this lesson, you will be able to turn a question into a vector, compare it with the vectors of all chunks, and pick the chunks most similar to the question. You will use cosine similarity, a common way to measure how alike two vectors are.

From chunks to vectors

In the previous lessons, you split a document into chunks and turned each chunk into a vector of character trigram counts. Now you need a way to compare a question with those chunks. The question is also text, so you can turn it into a vector using the same vocabulary. Then you measure how similar the question vector is to each chunk vector.

You already have a file called vectorizer.py with the functions make_trigrams, build_vocabulary, and chunk_to_vector. You will reuse them here. If you have not done the exercise from lesson 2, create vectorizer.py now with this code:

import re

def make_trigrams(text):
    text = ' ' + text.lower() + ' '
    return [text[i:i+3] for i in range(len(text)-2)]

def build_vocabulary(chunks):
    vocab = set()
    for chunk in chunks:
        trigrams = make_trigrams(chunk)
        vocab.update(trigrams)
    return sorted(vocab)

def chunk_to_vector(chunk, vocab):
    trigrams = make_trigrams(chunk)
    counts = {}
    for tri in trigrams:
        counts[tri] = counts.get(tri, 0) + 1
    vector = []
    for tri in vocab:
        vector.append(counts.get(tri, 0))
    return vector

Cosine similarity

Cosine similarity measures the angle between two vectors. If the vectors point in the same direction, the similarity is 1. If they are perpendicular, the similarity is 0. If they point in opposite directions, the similarity is -1. For our vectors, which contain only non-negative counts, the similarity will be between 0 and 1.

The formula for cosine similarity between two vectors A and B is:

similarity = (A dot B) / (||A|| * ||B||)

Here A dot B is the sum of the products of corresponding entries. ||A|| is the length of vector A, calculated as the square root of the sum of the squares of its entries.

You can write a function to compute cosine similarity in Python. Create a new file called retriever.py:

import math

def cosine_similarity(vec_a, vec_b):
    dot = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = math.sqrt(sum(a * a for a in vec_a))
    norm_b = math.sqrt(sum(b * b for b in vec_b))
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)

The function takes two vectors of the same length. It computes the dot product, then the norms, and returns the similarity. If either vector is all zeros, it returns 0 to avoid division by zero.

Turn a question into a vector

To compare a question with chunks, you need to turn the question into a vector using the same vocabulary as the chunks. Add this function to retriever.py:

from vectorizer import make_trigrams, chunk_to_vector

def question_to_vector(question, vocab):
    return chunk_to_vector(question, vocab)

This function simply calls chunk_to_vector with the question text. The vocabulary must be the one built from the document chunks.

Rank chunks by similarity

Now you can compute the similarity between the question vector and each chunk vector. You will store the similarity along with the chunk text and its index, then sort them in descending order. Add this function to retriever.py:

def retrieve_top_chunks(question, chunks, vocab, top_k=3):
    q_vec = question_to_vector(question, vocab)
    scored = []
    for i, chunk in enumerate(chunks):
        c_vec = chunk_to_vector(chunk, vocab)
        sim = cosine_similarity(q_vec, c_vec)
        scored.append((sim, i, chunk))
    scored.sort(reverse=True, key=lambda x: x[0])
    return scored[:top_k]

The function takes the question, the list of chunks, the vocabulary, and an optional top_k (default 3). It builds the question vector, then loops over each chunk, computes its vector, and calculates the cosine similarity. It stores a tuple with the similarity, the chunk index, and the chunk text. After sorting in descending order by similarity, it returns the top top_k tuples.

Test the retriever

Create a small script to test the retriever. You can use the same sample text from lesson 1. Create a file called test_retriever.py:

from vectorizer import build_vocabulary
from retriever import retrieve_top_chunks

def read_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as f:
        return f.read()

def chunk_words(words, chunk_size, overlap):
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = words[start:end]
        chunks.append(' '.join(chunk))
        start += chunk_size - overlap
    return chunks

def main():
    text = read_file('sample.txt')
    words = text.split()
    chunks = chunk_words(words, 10, 5)
    vocab = build_vocabulary(chunks)
    question = 'What is retrieval-augmented generation?'
    top = retrieve_top_chunks(question, chunks, vocab, top_k=2)
    for sim, idx, chunk in top:
        print(f'Similarity: {sim:.4f}')
        print(f'Chunk {idx+1}: {chunk}')
        print()

if __name__ == '__main__':
    main()

This script reads sample.txt, chunks it with size 10 and overlap 5, builds the vocabulary, and retrieves the top 2 chunks for a sample question. Run it:

python test_retriever.py

You will see the similarity scores and the chunk texts. The chunks most similar to the question should appear first. The exact scores depend on the trigrams in the text, but the order is what matters.

Choosing the number of chunks

The parameter top_k controls how many chunks you pass to the generator later. A small number keeps the prompt short, but you might miss relevant information. A large number gives more context but may include noise. In the next lessons, you will use the retrieved chunks to build a prompt. For now, top_k=3 is a reasonable default.

You have now built the retriever. In the next lesson, you will combine the question and the retrieved chunks into a prompt for a language model.

Key takeaways

  • Cosine similarity measures the angle between two vectors; for non-negative vectors it ranges from 0 to 1.
  • Turn the question into a vector using the same vocabulary as the chunks.
  • Rank chunks by cosine similarity and select the top few.
  • The number of chunks to retrieve is a parameter you can tune.

Try it

Write a Python script that reads a plain-text file of your choice, splits it into chunks of 50 words with 10 words overlap, builds the vocabulary, and retrieves the top 3 chunks for a question you make up. Print the similarity score and the chunk text for each. Then, change the question to something unrelated to the document and run again. A good result shows that the top chunks for a relevant question have higher similarity scores than those for an unrelated question, and the chunks are printed in descending order of similarity.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.

    Retrieve the Most Relevant Chunks | Build a Retrieval-Augmented Generation System in Python | Rolebound