Introduction to the Tidyverse

Week 7, Part 2

Jeff Jacobs

jj1088@georgetown.edu
DSAN Summer Bootcamp 2024
Georgetown University

Overview

  • What is “tidy” data? (Next slide)
  • Tidyverse = “universe” of add-on libraries for R, which assume tidy data so as to streamline data cleaning/analysis!

Tidy Data

  • Every row represents an observation
  • Every column represents an attribute
  • Every cell contains a value \(V_{{\color{#E69F00}i}{\color{#56B4E9}j}}\) = attribute \(j\) for observation \(i\)
Var A Var B
Obs 1 Val 1A Val 1B
Obs 2 Val 2A Val 2B
Table 1: A tidy dataset, with observation IDs in orange, column names in blue, and values in green

✅ Tidy!
id name flights
DCA Reagan National 11.55
BWI Baltimore-Wash Int'l 11.15
IAD Dulles International 10.27

Tidy Data Check

  1. Choose a row, take note of its ID
  2. Go through values in row, asking yourself: is this the value of some attribute of [observation]?
❌ Not Tidy!
id var value
DCA name Reagan National
DCA flights 11.55
BWI name Baltimore-Wash Int'l
BWI flights 11.15
IAD name Dulles International
IAD flights 10.27

Observation IDs Can Be Tuples!

library(tidyverse)
table1
country year cases population
Afghanistan 1999 745 19987071
Afghanistan 2000 2666 20595360
Brazil 1999 37737 172006362
Brazil 2000 80488 174504898
China 1999 212258 1272915272
China 2000 213766 1280428583

✅ Tidy! Here we pass the “Tidy Data Check” since 745 is a property of [Afghanistan in 1999]: its number of TB cases

library(tidyverse)
table2 |> head()
country year type count
Afghanistan 1999 cases 745
Afghanistan 1999 population 19987071
Afghanistan 2000 cases 2666
Afghanistan 2000 population 20595360
Brazil 1999 cases 37737
Brazil 1999 population 172006362

❌ Not Tidy! Here we fail the Tidy Data Check since cases is not a property of [Afghanistan in 1999]

So… How Do We Get Our Data Into Tidy Form?

G cluster_00 Data-Processing Pipeline 1 raw Raw Data tr1 Transformation A (select(), filter()) raw->tr1 tr2 Transformation B (mutate(), summarize()) tr1->tr2 viz Visualization tr2->viz

G cluster_00 Data-Processing Pipeline 2 raw Raw Data tr1 Transformation C (select(), filter()) raw->tr1 tr2 Transformation D (mutate(), summarize()) tr1->tr2 viz      Result      tr2->viz

raw_data |> select() |> mutate() |> visualize()
raw_data |> filter() |> summarize() |> check_result()

The Tidyverse


dplyr

ggplot2

forcats

tibble

readr

stringr

tidyr

purrr

readr

Homepage | Overview (R4DS) | PDF Cheatsheet

Two key functions: read_csv(), write_csv() (Bonus: read_delim() if others fail)

Warning! read_csv() vs. read.csv()

Note that these are not the same as R’s built-in read.csv() and write.csv()! The built-in R functions will produce a plain data.frame object, not a tibble

  • Can handle URLs as well!
  • Share data+code in seconds by using in combination with GitHub Gist
  • (No read_csv() path issues either 😉)
  • Gist dataset
gdp_df <- read_csv("https://gist.githubusercontent.com/jpowerj/fecd437b96d0954893de727383f2eaf2/raw/fec58507f7095cb8341b229d6eb74ce53232d663/gdp_2010.csv")
gdp_df |> head(6)
name code year value
Afghanistan AFG 2010 15936800636
Albania ALB 2010 11926953259
Algeria DZA 2010 161207268655
American Samoa ASM 2010 576000000
Andorra AND 2010 3355695364
Angola AGO 2010 82470913121

tibble

Homepage | Overview (R4DS) | PDF Cheatsheet

Replaces R’s built-in data.frame, but retains syntax for backwards compatibility:

library(tibble)
data <- c(3.4,1.1,9.6)
labels <- c(0,1,0)
supervised_df <- tibble::tibble(
  x=data,
  y=labels
)
supervised_df
x y
3.4 0
1.1 1
9.6 0
  • Provides a surprisingly useful function: tribble() (tibble defined row-by-row)
library(tibble)
dsan_df <- tibble::tribble(
    ~code, ~topic, ~credits,
    "dsan5000", "Data Science", 3,
    "dsan5100", "Probability", 3,
    "dsan5200", "Visualization", 3
)
dsan_df
code topic credits
dsan5000 Data Science 3
dsan5100 Probability 3
dsan5200 Visualization 3

dplyr

Homepage | Overview (R4DS) | PDF Cheatsheet

Grammar of data manipulation (verbs):

filter()

select()

arrange()

mutate()

summarize()

filter(): Keep rows matching a boolean condition

table1 |> filter(year == 2000)
country year cases population
Afghanistan 2000 2666 20595360
Brazil 2000 80488 174504898
China 2000 213766 1280428583

dplyr

select():

table1 |> select(country)
country
Afghanistan
Afghanistan
Brazil
Brazil
China
China
table1 |> select(-country) |> head(3)
year cases population
1999 745 19987071
2000 2666 20595360
1999 37737 172006362

arrange():

table1 |> arrange(population)
country year cases population
Afghanistan 1999 745 19987071
Afghanistan 2000 2666 20595360
Brazil 1999 37737 172006362
Brazil 2000 80488 174504898
China 1999 212258 1272915272
China 2000 213766 1280428583
table1 |> arrange(desc(population)) |> head(3)
country year cases population
China 2000 213766 1280428583
China 1999 212258 1272915272
Brazil 2000 80488 174504898

dplyr

mutate():

table1 |> mutate(newvar = 300)
country year cases population newvar
Afghanistan 1999 745 19987071 300
Afghanistan 2000 2666 20595360 300
Brazil 1999 37737 172006362 300
Brazil 2000 80488 174504898 300
China 1999 212258 1272915272 300
China 2000 213766 1280428583 300

dplyr

summarize():

Code
table1 |> 
  summarize(
    avg_cases = mean(cases),
    avg_pop = mean(population)
  )
avg_cases avg_pop
91276.67 490072924

summarize() with grouping:

Code
table1 |>
  group_by(country) |>
  summarize(
    avg_cases = mean(cases),
    avg_pop = mean(population)
  )
country avg_cases avg_pop
Afghanistan 1705.5 20291216
Brazil 59112.5 173255630
China 213012.0 1276671928

The Tidyverse in Action

  • Let’s walk through a common data-analysis task (data cleaning and merging), seeing how the readr, tibble and dplyr functions help us!

Merging Data

  • The task: Analyze relationship between population and GDP (in 2000)
  • The data: One dataset on population in 2000, another on GDP in 2000 and 2010
  • Let’s get the data ready for merging using R
df <- table1 |>
  select(country, year, population) |>
  filter(year == 2000)
df |> write_csv("assets/pop_2000.csv")
df
country year population
Afghanistan 2000 20595360
Brazil 2000 174504898
China 2000 1280428583
gdp_df <- read_csv("https://gist.githubusercontent.com/jpowerj/c83e87f61c166dea8ba7e4453f08a404/raw/29b03e6320bc3ffc9f528c2ac497a21f2d801c00/gdp_2000_2010.csv")
gdp_df |> head(5)
Country Name Country Code Year Value
Afghanistan AFG 2010 15936800636
Albania ALB 2000 3632043908
Albania ALB 2010 11926953259
Algeria DZA 2000 54790245601
Algeria DZA 2010 161207268655

Selecting/Filtering in Action

Let’s keep just the rows containing data from 2000, drop extraneous columns, and then rename the remaining columns to get rid of the spaces

gdp_2000_df <- gdp_df |>
  select(`Country Name`,Year,Value) |>
  filter(Year == "2000") |>
  rename(country=`Country Name`, year=`Year`, gdp=`Value`)
gdp_2000_df |> write_csv("assets/gdp_2000.csv")
gdp_2000_df |> head()
country year gdp
Albania 2000 3632043908
Algeria 2000 54790245601
Andorra 2000 1434429703
Angola 2000 9129594819
Antigua and Barbuda 2000 830158769
Argentina 2000 284203750000

Switching to Python for Merging!

  • Remember that we’re using Quarto, so we can switch to Python whenever we’d like!
  • Pandas provides an easy-to-use df.merge(other_df), let’s use it 😎
import pandas as pd
from IPython.display import Markdown
pop_df = pd.read_csv("assets/pop_2000.csv")
gdp_df = pd.read_csv("assets/gdp_2000.csv")
Left Join
merged_df = pop_df.merge(gdp_df,
  on='country', how='left', indicator=True
)
Markdown(merged_df.to_markdown())
country year_x population year_y gdp _merge
0 Afghanistan 2000 20595360 nan nan left_only
1 Brazil 2000 174504898 2000 6.55421e+11 both
2 China 2000 1280428583 2000 1.21135e+12 both
Inner join (≈ Intersection (\(\cap\)))
merged_df = pop_df.merge(gdp_df,
  on='country', how='inner', indicator=True
)
Markdown(merged_df.to_markdown())
country year_x population year_y gdp _merge
0 Brazil 2000 174504898 2000 6.55421e+11 both
1 China 2000 1280428583 2000 1.21135e+12 both

(see Appendix Slides → for merging via Tidyverse)

Reshaping Data

Sometimes you can’t merge because one of the datasets looks like the table on the left (non-tidy), but we need it to look like the table on the right (tidy)…

In data-cleaning jargon, this dataset is long (more than one row per observation)

table2 |> write_csv("assets/long_data.csv")
table2 |> head()
country year type count
Afghanistan 1999 cases 745
Afghanistan 1999 population 19987071
Afghanistan 2000 cases 2666
Afghanistan 2000 population 20595360
Brazil 1999 cases 37737
Brazil 1999 population 172006362

In data-cleaning jargon, this dataset is wide (one row per obs; usually tidy)

table1 |> write_csv("assets/wide_data.csv")
table1 |> head()
country year cases population
Afghanistan 1999 745 19987071
Afghanistan 2000 2666 20595360
Brazil 1999 37737 172006362
Brazil 2000 80488 174504898
China 1999 212258 1272915272
China 2000 213766 1280428583

Long-to-Wide: pivot_wider()

pivot_wider() Documentation (part of tidyr)

table2 |> head()
country year type count
Afghanistan 1999 cases 745
Afghanistan 1999 population 19987071
Afghanistan 2000 cases 2666
Afghanistan 2000 population 20595360
Brazil 1999 cases 37737
Brazil 1999 population 172006362
wide_df <- table2 |> pivot_wider(
  id_cols = c(country, year),
  names_from = type,
  values_from = count
)
wide_df
country year cases population
Afghanistan 1999 745 19987071
Afghanistan 2000 2666 20595360
Brazil 1999 37737 172006362
Brazil 2000 80488 174504898
China 1999 212258 1272915272
China 2000 213766 1280428583

Wide-to-Long: pivot_longer()

pivot_longer() Documentation (part of tidyr)

wide_df |> head()
country year cases population
Afghanistan 1999 745 19987071
Afghanistan 2000 2666 20595360
Brazil 1999 37737 172006362
Brazil 2000 80488 174504898
China 1999 212258 1272915272
China 2000 213766 1280428583
long_df <- wide_df |> pivot_longer(
  cols = c(cases, population),
  names_to = "var",
  values_to = "val"
)
long_df |> head()
country year var val
Afghanistan 1999 cases 745
Afghanistan 1999 population 19987071
Afghanistan 2000 cases 2666
Afghanistan 2000 population 20595360
Brazil 1999 cases 37737
Brazil 1999 population 172006362

Appendix: The Rest of the Tidyverse

  • forcats
  • readr
  • stringr
  • tidyr
  • purrr
  • lubridate*

forcats

Homepage | Overview | Cheatsheet

  • Utilities for working with factor variables (R’s data structure for categorical variables)
  • factors = data + levels:
Code
month_levels <- c(
    "Jan", "Feb", "Mar", "Apr",
    "May", "Jun", "Jul", "Aug",
    "Sep", "Oct", "Nov", "Dec"
)
d <- c("Jan","Jan","Feb","Dec")
print(d)
dataf <- parse_factor(
    d,
    levels=month_levels
)
print(dataf)
[1] "Jan" "Jan" "Feb" "Dec"
[1] Jan Jan Feb Dec
Levels: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Factors for Ordering Plot Elements

relig_summary <- gss_cat %>%
  group_by(relig) %>%
  summarise(
    age = mean(age, na.rm = TRUE),
    tvhours = mean(tvhours, na.rm = TRUE),
    n = n()
  )
relig_labs <- labs(
  x = "TV Hours / Day",
  y = "Religion"
)

Without forcats:

Code
relig_summary |>
  ggplot(aes(tvhours, relig)) +
  geom_point(size=g_pointsize) +
  geom_segment(aes(yend = relig, x=0, xend = tvhours)) +
  dsan_theme("half") +
  relig_labs

With forcats:

Code
relig_summary |>
  mutate(relig = fct_reorder(relig, tvhours)) |>
  ggplot(aes(x=tvhours, y=relig)) +
    geom_point(size=g_pointsize) +
    geom_segment(aes(yend = relig, x=0, xend = tvhours)) +
    dsan_theme("half") +
    relig_labs

Sorting Barplots with fct_infreq()

barplot_labs <- labs(
  title = "Respondents by Marital Status",
  x = "Marital Status",
  y = "Count"
)
Code
gss_cat |>
  mutate(marital = marital |> fct_infreq() |> fct_rev()) |>
  ggplot(aes(marital)) + geom_bar() + barplot_labs +
  dsan_theme("full")

Recoding/Combining Categories

Automatically combining using fct_lump():

Code
gss_cat |>
    mutate(relig = fct_lump(relig)) |>
    count(relig)
relig n
Protestant 10846
Other 10637

Manually Combining using fct_recode():

Code
gss_cat |>
    mutate(partyid = fct_recode(partyid,
    "Republican"  = "Strong republican",
    "Republican"  = "Not str republican",
    "Independent" = "Ind,near rep",
    "Independent" = "Ind,near dem",
    "Democrat"    = "Not str democrat",
    "Democrat"    = "Strong democrat",
    "Other"       = "No answer",
    "Other"       = "Don't know",
    "Other"       = "Other party"
  )) |>
  count(partyid)
partyid n
Other 548
Republican 5346
Independent 8409
Democrat 7180

purrr: Functional Programming

Homepage | Overview | Cheatsheet | Tutorials, with Applications

Provides an anonymous function operator ~, arguments get named .x, .y:

Code
my_points <- c("Midterm"=18, "Final"=300)
total_points <- c("Midterm"=20, "Final"=400)
(avg_score <- map2(my_points, total_points,
  ~ list(frac=.x / .y, pct=(.x/.y)*100)))
$Midterm
$Midterm$frac
[1] 0.9

$Midterm$pct
[1] 90


$Final
$Final$frac
[1] 0.75

$Final$pct
[1] 75

Along with helpful functions for transforming the output

Code
list_flatten(avg_score)
$Midterm_frac
[1] 0.9

$Midterm_pct
[1] 90

$Final_frac
[1] 0.75

$Final_pct
[1] 75
Code
every(avg_score, ~ .x$frac > 0.5)
[1] TRUE

lubridate*

Homepage | Overview | Cheatsheet

Caution: Importing lubridate

Note that lubridate does not get loaded with the other packages in the tidyverse when you call library(tidyverse). It needs to be imported explicitly:

library(tidyverse)
library(lubridate)

Appendix 2: Wide-to-Long in Base R with gather()

Code
table1
country year cases population
Afghanistan 1999 745 19987071
Afghanistan 2000 2666 20595360
Brazil 1999 37737 172006362
Brazil 2000 80488 174504898
China 1999 212258 1272915272
China 2000 213766 1280428583
Code
long_df <- gather(table1,
  key = "variable",
  value = cases,
  -c(country, year)
)
long_df |> head()
country year variable cases
Afghanistan 1999 cases 745
Afghanistan 2000 cases 2666
Brazil 1999 cases 37737
Brazil 2000 cases 80488
China 1999 cases 212258
China 2000 cases 213766