Midterm → Final Project Datasets

Selection Criteria

Looking at a dataset of all H1-B visa job registrations submitted by US employers to the Department of Labor, the top industries (by share of total postings per NAICS code) hiring data scientists specifically (OES Code 15-2051) are:

Rank	Industry Group	NAICS Code	Percent
1	Tech Companies	5415	21.44%
2	Banking and Finance	52	7.95%
3	Retail Sales	45	7.72%
4	Consulting Firms	5416	4.01%
5	Scientific Research Labs	5417	2.58%
6	Healthcare Organizations	62	2.06%

Thus, in choosing datasets for the midterm, my aim was to cover the types of data you might encounter working as a data scientist / engineer within each of these (admittedly very broadly-defined) areas!

One implication of this approach is that, although some of the data is historical (like the Enron Email Corpus), the goal will not be just to analyze this nicely-packaged self-contained datasets, but instead to imagine how you would handle new data continuously streaming into an OLTP database with the same structure! So, in the Enron Email Corpus example, your interest won’t be in analyzing those emails in particular, but to use them as a case study, extrapolating how you might use the tools we’ve learned in class thus far to derive analytic insights from a hypothetical employer’s email database.

Datasets with Midterm Vignettes

Additional Final Project Datasets

Like the above datasets, these are pre-approved if you find them interesting enough to analyze for your Final Project! They do not have midterm vignettes, but you can start diving into them immediately after handing in your midterm if you’d like 😎