Skip to content
Free courses
Data EngineeringBeginnerFree

Build a Small Data Pipeline with a Warehouse and Lake

Create a small data pipeline that loads raw CSV files into a SQL warehouse, cleans and joins them, and exports a summary to a data lake. You will automate the whole run with a simple script.

6 lessons 45 minNo account needed

Start lesson 1

What you will learn

  • Describe the difference between a data lake and a data warehouse.
  • Build a small data pipeline that loads raw CSV files into a SQL warehouse.
  • Clean and join data with SQL to create analysis-ready tables.
  • Export a summary from a warehouse to a data lake as a CSV file.
  • Write a simple script that runs the full pipeline in order.

Before you start

  • Use a text editor to create and save plain text files.
  • Open a command line on your computer.
  • Understand rows and columns in a spreadsheet.

Lessons

  1. Lesson 1: Create a Data Lake and Add Raw CSV FilesSet up a project folder with a raw-data subfolder and add three small CSV files that act as a data lake.6 min
  2. Lesson 2: Set Up a Warehouse with SQLite TablesCreate a SQLite database file and define tables for customers, products, and orders with primary keys and foreign keys.7 min
  3. Lesson 3: Load Raw Files into Staging TablesUse the SQLite command-line tool to import each CSV file into a staging table inside the warehouse.10 min
  4. Lesson 4: Clean and Join Data into an Orders TableWrite SQL to remove blank values, join staging tables, and insert clean rows into a final orders table.8 min
  5. Lesson 5: Export a Summary to the LakeQuery the warehouse to create a monthly sales summary and export it as a CSV file back into the data lake.8 min
  6. Lesson 6: Automate the Whole Pipeline with a ScriptPut the import, clean, join, and export steps into one shell script that you can run end to end.6 min
8,237 open data engineer roles on Rolebound right nowSee the roles

Ready to use this in a job?

Rolebound applies to matching roles for you.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.