Skip to content

Lesson 2 of 6

Represent Chunks as Vectors with Character N-grams

Convert each chunk into a fixed-length numerical vector using character n-gram frequencies, so chunks can be compared mathematically.

6 minAI Engineering

After this lesson, you will be able to turn a text chunk into a fixed-length vector of numbers. You will use character n-grams, which are short overlapping sequences of characters, to capture the chunk's content. These vectors will let you compare chunks mathematically in the next lesson.

Why vectors?

A computer cannot compare two pieces of text directly. It needs numbers. A vector is a list of numbers that represents something, in this case a chunk of text. If two chunks are about similar things, their vectors should be similar. In the next lesson, you will measure that similarity to retrieve relevant chunks.

Character n-grams

An n-gram is a sequence of n items. A character n-gram is a sequence of n characters. For example, the word 'cat' has these 2-grams (also called bigrams): 'ca' and 'at'. The word 'dog' has 'do' and 'og'. Character n-grams are useful because they capture spelling patterns and are robust to small differences in wording.

We will use n=3, so trigrams. For the chunk 'hello', the trigrams are 'hel', 'ell', and 'llo'. We add spaces at the start and end of the text so that the first and last characters appear in the same number of trigrams as the middle characters. With padding, ' hello ' gives ' he', 'hel', 'ell', 'llo', 'lo '.

Building the vocabulary

We need a fixed list of all possible trigrams so that every chunk gets a vector of the same length. The vocabulary is the set of all trigrams that appear in the document. We build it once from the whole document, then use it for every chunk.

Start with the code from the previous lesson. You have a function read_file that reads a text file, and a function chunk_words that splits the text into chunks. Create a new file called vectorizer.py and add the following code:

import re

def make_trigrams(text):
    text = ' ' + text.lower() + ' '
    return [text[i:i+3] for i in range(len(text)-2)]

def build_vocabulary(chunks):
    vocab = set()
    for chunk in chunks:
        trigrams = make_trigrams(chunk)
        vocab.update(trigrams)
    return sorted(vocab)

The function make_trigrams takes a string, converts it to lowercase, adds spaces at both ends, and returns a list of all three-character substrings. The function build_vocabulary takes a list of chunks, collects all trigrams from all chunks into a set to remove duplicates, and returns a sorted list. Sorting makes the order deterministic, which is important for consistent vectors.

Creating vectors

A vector for a chunk is a list of numbers, one for each trigram in the vocabulary. The number is the frequency of that trigram in the chunk: how many times it appears. For example, if the vocabulary is ['ell', 'hel', 'lo ', 'llo'] and the chunk is 'hello', the frequencies are: 'ell' appears once, 'hel' once, 'lo ' once, 'llo' once. So the vector is [1, 1, 1, 1].

Add this function to vectorizer.py:

def chunk_to_vector(chunk, vocab):
    trigrams = make_trigrams(chunk)
    counts = {}
    for tri in trigrams:
        counts[tri] = counts.get(tri, 0) + 1
    vector = []
    for tri in vocab:
        vector.append(counts.get(tri, 0))
    return vector

The function chunk_to_vector takes a chunk and the vocabulary. It makes the trigrams for the chunk, counts how many times each trigram appears using a dictionary, then builds a list by looking up each vocabulary trigram in the counts dictionary. If a trigram is not in the chunk, its count is 0.

Testing the vectorizer

Let's test with a small example. Create a file called test_vectorizer.py with this code:

from vectorizer import make_trigrams, build_vocabulary, chunk_to_vector

chunks = ['hello world', 'world hello']
vocab = build_vocabulary(chunks)
print('Vocabulary:', vocab)
for chunk in chunks:
    vec = chunk_to_vector(chunk, vocab)
    print(f'Chunk: {chunk}')
    print(f'Vector: {vec}')

Run it with python test_vectorizer.py. The vocabulary will be a sorted list of all trigrams from both chunks. Each vector will have the same length as the vocabulary. The two chunks have the same words in different order, so their vectors will be different, but they will share many non-zero entries because they contain the same trigrams.

Using the vectorizer with your chunks

Now integrate the vectorizer into your chunking script. Modify chunker.py so that after creating the chunks, it builds the vocabulary and prints the vector for each chunk. Here is the full updated chunker.py:

from vectorizer import build_vocabulary, chunk_to_vector

def read_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as f:
        return f.read()

def chunk_words(words, chunk_size, overlap):
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = words[start:end]
        chunks.append(' '.join(chunk))
        start += chunk_size - overlap
    return chunks

def main(file_path, chunk_size, overlap):
    text = read_file(file_path)
    words = text.split()
    chunks = chunk_words(words, chunk_size, overlap)
    vocab = build_vocabulary(chunks)
    for i, chunk in enumerate(chunks):
        vec = chunk_to_vector(chunk, vocab)
        print(f'Chunk {i+1}: {chunk}')
        print(f'Vector (first 10 values): {vec[:10]}')
        print()

if __name__ == '__main__':
    import sys
    if len(sys.argv) != 4:
        print('Usage: python chunker.py <file> <chunk_size> <overlap>')
        sys.exit(1)
    main(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]))

Run it with python chunker.py sample.txt 10 5. You will see each chunk followed by the first ten values of its vector. The full vector is longer, but printing only the first ten keeps the output readable. The vector length is the number of trigrams in the vocabulary, which depends on the document.

Why this works

Character n-gram vectors capture the surface form of text. Chunks that share many character sequences will have similar vectors. This is a simple and fast way to represent text for retrieval. In the next lesson, you will use these vectors to find the chunks most similar to a question.

Key takeaways

  • A vector is a list of numbers that represents a chunk of text.
  • Character n-grams are short overlapping sequences of characters, and we use trigrams (n=3).
  • The vocabulary is the sorted set of all trigrams in the document, giving every vector the same length.
  • A chunk's vector contains the frequency of each vocabulary trigram in that chunk.
  • These vectors allow mathematical comparison of chunks, which is the basis for retrieval.

Try it

Write a Python script that reads a plain-text file, splits it into chunks of 100 words with 20 words overlap, builds the vocabulary, and prints the vector for the first chunk. Then, manually verify that the vector length equals the number of trigrams in the vocabulary. Choose a small file (a few paragraphs) so you can inspect the output. A good result shows a vector of integers, all non-negative, with length equal to the vocabulary size, and the sum of the vector entries equals the number of trigrams in that chunk (including duplicates).

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.

    Represent Chunks as Vectors with Character N-grams | Build a Retrieval-Augmented Generation System in Python | Rolebound