Lesson 5 of 6
Generate an Answer with a Placeholder Language Model
Use a simple function that stands in for a language model to produce an answer from the prompt, so you can test the full pipeline.
6 minAI Engineering
After this lesson, you will be able to use a placeholder language model to generate an answer from a prompt, and test the full retrieval-augmented generation pipeline from document to answer.
Why a placeholder?
A real language model is a large neural network that takes a prompt and produces text. For this course, we use a simple function that mimics that behaviour. It lets you test the whole pipeline without needing an internet connection or a large model. Later, you can replace this placeholder with a real model.
The placeholder model works by looking at the context you give it. It finds the sentence in the context that is most similar to the question, and returns that sentence as the answer. This is a simple rule, but it is enough to see how the pieces fit together.
Create the placeholder model
Create a new file called generator.py. In this file, write a function called generate_answer that takes a prompt and returns an answer string.
The prompt has a fixed structure from the earlier lessons. It starts with an instruction, then a context section with numbered chunks, then the question, and finally the word Answer:. Because the chunks themselves are separated by double newlines, we cannot split the whole prompt on double newlines. Instead, we find the markers Context:\n and \n\nQuestion: to extract the question and the full context.
Here is the code for generator.py:
import re
def generate_answer(prompt):
context_marker = 'Context:\n'
question_marker = '\n\nQuestion: '
start = prompt.index(context_marker) + len(context_marker)
end = prompt.index(question_marker)
context_text = prompt[start:end]
question = prompt[end + len(question_marker):].split('\nAnswer:')[0].strip()
raw_chunks = context_text.split('\n\n')
chunks = []
for raw in raw_chunks:
cleaned = re.sub(r'^\[\d+\]\s*', '', raw.strip())
cleaned = cleaned.strip()
if cleaned:
chunks.append(cleaned)
question_words = set(re.findall(r'\w+', question.lower()))
best_chunk = ''
best_score = -1
for chunk in chunks:
chunk_words = set(re.findall(r'\w+', chunk.lower()))
score = len(question_words.intersection(chunk_words))
if score > best_score:
best_score = score
best_chunk = chunk
if best_score == 0:
return ''
return best_chunkThis function does the following:
- It finds the context between
Context:\nand\n\nQuestion:. - It splits the context into individual chunks by double newlines.
- It removes the leading
[1],[2]and similar prefixes from each chunk. - It turns the question and each chunk into sets of words. The tokeniser ignores punctuation, so
generation?andgenerationmatch. - It counts how many words the question and chunk have in common, and chooses the chunk with the highest score.
- If no words match, it returns an empty string.
This is a very simple model, but it works for testing.
Test the generator with a sample prompt
Create a file called test_generator.py with the following code. It uses build_prompt from the earlier lessons to create a prompt with the exact format the pipeline will use.
from prompt_builder import build_prompt
from generator import generate_answer
retrieved = [
(0.9, 0, 'Retrieval-augmented generation combines a retriever and a generator.'),
(0.8, 1, 'The retriever finds relevant passages from a document.')
]
question = 'What is retrieval-augmented generation?'
prompt = build_prompt(question, retrieved)
answer = generate_answer(prompt)
print('Answer:', answer)Run the script:
python test_generator.pyThe output should be:
Answer: Retrieval-augmented generation combines a retriever and a generator.The placeholder model picked the first chunk because it shares several content words with the question, such as retrieval, augmented and generation.
Connect the generator to the full pipeline
Now you can combine all the pieces from the earlier lessons: reading a file, chunking, vectorising, retrieving, building a prompt, and generating an answer.
Create a file called pipeline.py with the following code:
from vectorizer import build_vocabulary
from retriever import retrieve_top_chunks
from prompt_builder import build_prompt
from generator import generate_answer
def read_file(file_path):
with open(file_path, 'r', encoding='utf-8') as f:
return f.read()
def chunk_words(words, chunk_size, overlap):
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunk = words[start:end]
chunks.append(' '.join(chunk))
start += chunk_size - overlap
return chunks
def main(file_path, question, chunk_size=10, overlap=5, top_k=2):
text = read_file(file_path)
words = text.split()
chunks = chunk_words(words, chunk_size, overlap)
vocab = build_vocabulary(chunks)
top = retrieve_top_chunks(question, chunks, vocab, top_k=top_k)
prompt = build_prompt(question, top)
answer = generate_answer(prompt)
print('Answer:', answer)
if __name__ == '__main__':
import sys
if len(sys.argv) < 3:
print('Usage: python pipeline.py <file> <question>')
sys.exit(1)
main(sys.argv[1], sys.argv[2])This script takes a document file and a question as command-line arguments. It reads the file, splits it into chunks, builds the vocabulary, retrieves the top chunks, builds a prompt, and generates an answer using the placeholder model.
Run it with your own document, for example a file called sample.txt:
python pipeline.py sample.txt "What is retrieval-augmented generation?"The output will be the sentence from the document that the placeholder model thinks best answers the question. The exact answer depends on your document and the question.
Summary
You now have a complete retrieval-augmented generation pipeline. The placeholder model is simple, but it lets you test the flow from document to answer. In the next lesson, you will wrap this pipeline into a single command-line tool.
Key takeaways
- A placeholder language model can be a simple function that selects the most relevant sentence from the context.
- The generator takes the prompt, extracts the question and the numbered chunks, and returns one chunk as the answer.
- The full pipeline connects reading, chunking, vectorising, retrieving, prompting, and generating.
- You can test the pipeline with a document file and a question using the command-line script.
Try it
Write a Python script that reads a plain-text file of your choice, splits it into chunks of 30 words with 10 words overlap, builds the vocabulary, retrieves the top 3 chunks for a question you make up, builds a prompt, and generates an answer using the placeholder model. Print the prompt and the answer. Then, change the question to something unrelated to the document and run again. A good result shows that for a relevant question, the answer is a sentence from the document that shares content words with the question, and for an unrelated question, the answer is usually less relevant or may be empty. The prompt should show the retrieved chunks and the question.
Official documentation
Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.