Lesson 1 of 6
Split a Document into Overlapping Chunks
Read a plain-text file and split it into small overlapping passages so no meaning is lost at the edges.
7 minAI Engineering
After this lesson, you will be able to read a plain-text file in Python and split it into small overlapping passages called chunks. You will write a function that takes a long text and returns a list of chunks, each with a fixed number of words and a fixed overlap with the next chunk. This is the first step in building a retrieval-augmented generation system.
Why split a document?
A retrieval-augmented generation system answers a question by first finding relevant parts of a document and then using those parts to generate an answer. If the document is long, you cannot send the whole document to a language model every time. Instead, you split the document into smaller pieces, called chunks, and later retrieve only the chunks that are most relevant to the question.
Splitting by words is simple and works well for plain text. If you split into chunks of, say, 100 words, a sentence might be cut in the middle. To avoid losing meaning at the edges, you make chunks overlap. For example, chunk 1 contains words 1 to 100, chunk 2 contains words 51 to 150, chunk 3 contains words 101 to 200, and so on. The overlap is 50 words in this case.
Read a plain-text file
Create a new file called chunker.py. In this file, write a function that reads a text file and returns its content as a string.
def read_file(file_path):
with open(file_path, 'r', encoding='utf-8') as f:
return f.read()The open function opens the file. The 'r' mode means read. The encoding='utf-8' argument makes sure the file is read correctly even if it contains characters like é or ü. The with statement closes the file automatically when the block ends.
You can test this function with a small text file. Create a file called sample.txt with a few sentences, for example:
Retrieval-augmented generation combines a retriever and a generator. The retriever finds relevant passages from a document. The generator produces an answer based on those passages.Then run this in a Python shell or add a temporary print statement to your script:
print(read_file('sample.txt'))You should see the content of the file printed.
Split text into words
To split into chunks of a fixed number of words, you first need to turn the text into a list of words. Python's string method split() does this by splitting on whitespace.
text = "This is a simple example."
words = text.split()
print(words)This prints ['This', 'is', 'a', 'simple', 'example.']. Notice that the period stays attached to the word. For chunking, this is fine because we only need to count words.
Create overlapping chunks
Now write a function that takes a list of words and returns a list of chunks. Each chunk is a string of chunk_size words. Consecutive chunks overlap by overlap words.
def chunk_words(words, chunk_size, overlap):
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunk = words[start:end]
chunks.append(' '.join(chunk))
start += chunk_size - overlap
return chunksThe loop starts at index 0. It takes a slice of chunk_size words, joins them into a string, and appends it to the list. Then it moves the start forward by chunk_size - overlap. For example, if chunk_size is 100 and overlap is 50, the next start is 50, so the next chunk shares 50 words with the previous one. The loop stops when start is at or beyond the end of the word list.
The last chunk may be shorter than chunk_size if the text does not divide evenly. That is acceptable.
Put it together
Add a main function that reads a file, splits it into words, and prints the chunks. This lets you test the whole process from the command line.
def main(file_path, chunk_size, overlap):
text = read_file(file_path)
words = text.split()
chunks = chunk_words(words, chunk_size, overlap)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {chunk}\n")
if __name__ == '__main__':
import sys
if len(sys.argv) != 4:
print("Usage: python chunker.py <file> <chunk_size> <overlap>")
sys.exit(1)
main(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]))The if __name__ == '__main__' block runs only when you execute the script directly, not when you import it. It reads three command-line arguments: the file path, the chunk size, and the overlap. It converts the last two to integers because sys.argv gives strings.
Run the script with your sample file:
python chunker.py sample.txt 10 5You will see chunks of about 10 words, each overlapping the previous one by 5 words. The exact output depends on the number of words in your file.
Adjust chunk size and overlap
The right chunk size depends on your document and what you want to retrieve. A chunk should be large enough to contain a complete idea but small enough to be specific. A common starting point is 100 to 200 words. The overlap is often 10 to 20 percent of the chunk size. You will tune these values later when you test retrieval.
For now, keep the values as arguments so you can experiment without changing the code.
Key takeaways
- Split a document into overlapping chunks so that meaning is not lost at the edges.
- Read a plain-text file with Python's built-in
openfunction. - Use
str.split()to turn text into a list of words. - Write a loop that creates chunks of a fixed size with a fixed overlap.
- Make the chunk size and overlap configurable via command-line arguments.
Try it
Write a Python script that reads a plain-text file of your choice (for example, a public domain book from Project Gutenberg) and splits it into chunks of 150 words with an overlap of 25 words. Print the first three chunks and the total number of chunks. Check that the second chunk starts with the 126th word of the original text (since 150 - 25 = 125 words are skipped before the second chunk). A good result shows three chunks of about 150 words each, with the second chunk overlapping the first by 25 words, and the total number of chunks is roughly the number of words divided by 125.
Official documentation
Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.