Skip to content

Lesson 5 of 6

Generate an Answer with a Placeholder Language Model

Use a simple function that stands in for a language model to produce an answer from the prompt, so you can test the full pipeline.

6 minAI Engineering

After this lesson, you will be able to use a placeholder language model to generate an answer from a prompt, and test the full retrieval-augmented generation pipeline from document to answer.

Why a placeholder?

A real language model is a large neural network that takes a prompt and produces text. For this course, we use a simple function that mimics that behaviour. It lets you test the whole pipeline without needing an internet connection or a large model. Later, you can replace this placeholder with a real model.

The placeholder model works by looking at the context you give it. It finds the sentence in the context that is most similar to the question, and returns that sentence as the answer. This is a simple rule, but it is enough to see how the pieces fit together.

Create the placeholder model

Create a new file called generator.py. In this file, write a function called generate_answer that takes a prompt and returns an answer string.

The prompt has a fixed structure from the earlier lessons. It starts with an instruction, then a context section with numbered chunks, then the question, and finally the word Answer:. Because the chunks themselves are separated by double newlines, we cannot split the whole prompt on double newlines. Instead, we find the markers Context:\n and \n\nQuestion: to extract the question and the full context.

Here is the code for generator.py:

import re

def generate_answer(prompt):
    context_marker = 'Context:\n'
    question_marker = '\n\nQuestion: '
    start = prompt.index(context_marker) + len(context_marker)
    end = prompt.index(question_marker)
    context_text = prompt[start:end]
    question = prompt[end + len(question_marker):].split('\nAnswer:')[0].strip()
    raw_chunks = context_text.split('\n\n')
    chunks = []
    for raw in raw_chunks:
        cleaned = re.sub(r'^\[\d+\]\s*', '', raw.strip())
        cleaned = cleaned.strip()
        if cleaned:
            chunks.append(cleaned)
    question_words = set(re.findall(r'\w+', question.lower()))
    best_chunk = ''
    best_score = -1
    for chunk in chunks:
        chunk_words = set(re.findall(r'\w+', chunk.lower()))
        score = len(question_words.intersection(chunk_words))
        if score > best_score:
            best_score = score
            best_chunk = chunk
    if best_score == 0:
        return ''
    return best_chunk

This function does the following:

  • It finds the context between Context:\n and \n\nQuestion: .
  • It splits the context into individual chunks by double newlines.
  • It removes the leading [1], [2] and similar prefixes from each chunk.
  • It turns the question and each chunk into sets of words. The tokeniser ignores punctuation, so generation? and generation match.
  • It counts how many words the question and chunk have in common, and chooses the chunk with the highest score.
  • If no words match, it returns an empty string.

This is a very simple model, but it works for testing.

Test the generator with a sample prompt

Create a file called test_generator.py with the following code. It uses build_prompt from the earlier lessons to create a prompt with the exact format the pipeline will use.

from prompt_builder import build_prompt
from generator import generate_answer

retrieved = [
    (0.9, 0, 'Retrieval-augmented generation combines a retriever and a generator.'),
    (0.8, 1, 'The retriever finds relevant passages from a document.')
]
question = 'What is retrieval-augmented generation?'
prompt = build_prompt(question, retrieved)
answer = generate_answer(prompt)
print('Answer:', answer)

Run the script:

python test_generator.py

The output should be:

Answer: Retrieval-augmented generation combines a retriever and a generator.

The placeholder model picked the first chunk because it shares several content words with the question, such as retrieval, augmented and generation.

Connect the generator to the full pipeline

Now you can combine all the pieces from the earlier lessons: reading a file, chunking, vectorising, retrieving, building a prompt, and generating an answer.

Create a file called pipeline.py with the following code:

from vectorizer import build_vocabulary
from retriever import retrieve_top_chunks
from prompt_builder import build_prompt
from generator import generate_answer

def read_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as f:
        return f.read()

def chunk_words(words, chunk_size, overlap):
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = words[start:end]
        chunks.append(' '.join(chunk))
        start += chunk_size - overlap
    return chunks

def main(file_path, question, chunk_size=10, overlap=5, top_k=2):
    text = read_file(file_path)
    words = text.split()
    chunks = chunk_words(words, chunk_size, overlap)
    vocab = build_vocabulary(chunks)
    top = retrieve_top_chunks(question, chunks, vocab, top_k=top_k)
    prompt = build_prompt(question, top)
    answer = generate_answer(prompt)
    print('Answer:', answer)

if __name__ == '__main__':
    import sys
    if len(sys.argv) < 3:
        print('Usage: python pipeline.py <file> <question>')
        sys.exit(1)
    main(sys.argv[1], sys.argv[2])

This script takes a document file and a question as command-line arguments. It reads the file, splits it into chunks, builds the vocabulary, retrieves the top chunks, builds a prompt, and generates an answer using the placeholder model.

Run it with your own document, for example a file called sample.txt:

python pipeline.py sample.txt "What is retrieval-augmented generation?"

The output will be the sentence from the document that the placeholder model thinks best answers the question. The exact answer depends on your document and the question.

Summary

You now have a complete retrieval-augmented generation pipeline. The placeholder model is simple, but it lets you test the flow from document to answer. In the next lesson, you will wrap this pipeline into a single command-line tool.

Key takeaways

  • A placeholder language model can be a simple function that selects the most relevant sentence from the context.
  • The generator takes the prompt, extracts the question and the numbered chunks, and returns one chunk as the answer.
  • The full pipeline connects reading, chunking, vectorising, retrieving, prompting, and generating.
  • You can test the pipeline with a document file and a question using the command-line script.

Try it

Write a Python script that reads a plain-text file of your choice, splits it into chunks of 30 words with 10 words overlap, builds the vocabulary, retrieves the top 3 chunks for a question you make up, builds a prompt, and generates an answer using the placeholder model. Print the prompt and the answer. Then, change the question to something unrelated to the document and run again. A good result shows that for a relevant question, the answer is a sentence from the document that shares content words with the question, and for an unrelated question, the answer is usually less relevant or may be empty. The prompt should show the retrieved chunks and the question.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.