Week 6: Data Engineering
DSAN 6000: Big Data and Cloud Computing
Fall 2026
Data Engineering
- Have you noticed? In most HWs, you’ve looked at
.parquetmetadata describing a column schema - There’s a whole book (Reis and Housley 2022), and a whole job, focused directly on facilitating seamless “flow” from raw data \(\leadsto\) insights
- Today: Streamline separation and combination of (computation, data) by carefully defining and enforcing schemas
- Next week: Move to full-on distributed computing with Spark!
(…Much harder to debug issues with data flowing “out there” across EC2 nodes)
What is Data Engineering?
the white-eared puffbird book
Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning (Reis and Housley 2022)
Bolded terms \(\Rightarrow\) new skills, tools in your toolkit:
- Producers, consumers: Who is responsible for what?
- Data catalogs: How does someone in your org find the data they’re looking for, in their desired format?
- How is data made available? (Security, lifecycle management)
What Do Data Engineers Do?
- Disambiguation: How to interpret what the data means?
- Register the schema within a data catalog
- Subject matter experts (Data Stewards) add meaningful descriptions
- Deduplication: Having a single source of truth!
- Data Governance: How long to store, access controls, etc.
Once this is done, data may be stored in a central repository such as a data lake or data lakehouse.
Slide credit: Amit Arora!
OLTP \(\rightarrow\) OLAP: One Level Deeper
- Warehouses vs. Lakes vs. Lakehouses
- Star Schema
- Extract-Transform-Load (ETL) Pipelines
Reminder: Telemetry → Analytics
Recall how many tiny bits of information are streaming into a given app’s servers at any given time!
Database administrators therefore closely guard their OLTP databases. They are usually reluctant to let business analysts run ad hoc analytic queries on an OLTP database, since those queries are often expensive, scanning large parts of the dataset, which can harm the performance of concurrently executing transactions. (Kleppmann and Riccomini 2026)
Data Warehouse (Bill Inmon, 1989)
Before the data warehouse, the analysis would often occur directly on the source system itself, with the obvious consequence of bogging down production transactional databases with long-running queries. The goal of the data warehouse was to separate the source system [OLTP] from the analytical system [OLAP]. (Reis and Housley 2022, 295)
Data in the data warehouse is able to be used for many different purposes, including sitting and waiting for future requirements which are unknown today. (Inmon 1992)
Data Warehouse Principles
- Subject-oriented: Focused on a specific subject area (sales, marketing)
- Integrated: Data from disparate sources is consolidated and normalized
- Nonvolatile: Data remains unchanged once stored
- Time-variant: Varying time ranges can be queried
Kimball Star Schema
At center, a single fact table:
- Long, many many rows, as few columns as possible
- “Single source of truth”
Pointers to dimension tables:
- Wide, many many columns
- All relevant information about each value in the fact table
Data Lake
- As data scale grows, the “up front” work required to ETL data into warehouses becomes more and more daunting…
- Data lake = drop stringent requirements of warehouse (structured tables), and just… toss data into object storage (S3 buckets)!
- [Jeff’s take] Relates to schema-on-write vs. schema-on-read:
- Can’t write just anything into warehouse tables, needs to match schema
- …But, you can write just anything into S3 buckets
- Tossing stuff into a big lake can lead to data swamps 😱 …Gives rise to Data Lakehouse model (Databricks)
Data Lakehouse (Databricks)
- Still tossing data into “lake”
- But, data platforms like Databricks do extra work to organize it
- For example: Auto-scan the stuff tossed into the lake to populate data catalog
- …AWS Glue x AWS Athena = simplest possible version of this automated data cataloging!
AWS Athena: “Automated” ETL!
- Carry out analytic queries directly on S3 buckets!
What is the Quickest Possible Way to Explore Data in S3?
