Free courses
Data EngineeringBeginnerFree
Build a Small Data Pipeline with a Warehouse and Lake
Create a small data pipeline that loads raw CSV files into a SQL warehouse, cleans and joins them, and exports a summary to a data lake. You will automate the whole run with a simple script.
6 lessons 45 minNo account needed
Start lesson 1What you will learn
- Describe the difference between a data lake and a data warehouse.
- Build a small data pipeline that loads raw CSV files into a SQL warehouse.
- Clean and join data with SQL to create analysis-ready tables.
- Export a summary from a warehouse to a data lake as a CSV file.
- Write a simple script that runs the full pipeline in order.
Before you start
- Use a text editor to create and save plain text files.
- Open a command line on your computer.
- Understand rows and columns in a spreadsheet.
Lessons
- Lesson 1: Create a Data Lake and Add Raw CSV FilesSet up a project folder with a raw-data subfolder and add three small CSV files that act as a data lake.6 min
- Lesson 2: Set Up a Warehouse with SQLite TablesCreate a SQLite database file and define tables for customers, products, and orders with primary keys and foreign keys.7 min
- Lesson 3: Load Raw Files into Staging TablesUse the SQLite command-line tool to import each CSV file into a staging table inside the warehouse.10 min
- Lesson 4: Clean and Join Data into an Orders TableWrite SQL to remove blank values, join staging tables, and insert clean rows into a final orders table.8 min
- Lesson 5: Export a Summary to the LakeQuery the warehouse to create a monthly sales summary and export it as a CSV file back into the data lake.8 min
- Lesson 6: Automate the Whole Pipeline with a ScriptPut the import, clean, join, and export steps into one shell script that you can run end to end.6 min