Week 7, Part 2
jj1088@georgetown.edu
DSAN Summer Bootcamp 2024
Georgetown University
| Var A | Var B | |
|---|---|---|
| Obs 1 | Val 1A | Val 1B |
| Obs 2 | Val 2A | Val 2B |
| id | name | flights |
|---|---|---|
| DCA | Reagan National | 11.55 |
| BWI | Baltimore-Wash Int'l | 11.15 |
| IAD | Dulles International | 10.27 |
Tidy Data Check
| id | var | value |
|---|---|---|
| DCA | name | Reagan National |
| DCA | flights | 11.55 |
| BWI | name | Baltimore-Wash Int'l |
| BWI | flights | 11.15 |
| IAD | name | Dulles International |
| IAD | flights | 10.27 |
| country | year | cases | population |
|---|---|---|---|
| Afghanistan | 1999 | 745 | 19987071 |
| Afghanistan | 2000 | 2666 | 20595360 |
| Brazil | 1999 | 37737 | 172006362 |
| Brazil | 2000 | 80488 | 174504898 |
| China | 1999 | 212258 | 1272915272 |
| China | 2000 | 213766 | 1280428583 |
✅ Tidy! Here we pass the “Tidy Data Check” since 745 is a property of [Afghanistan in 1999]: its number of TB cases
| country | year | type | count |
|---|---|---|---|
| Afghanistan | 1999 | cases | 745 |
| Afghanistan | 1999 | population | 19987071 |
| Afghanistan | 2000 | cases | 2666 |
| Afghanistan | 2000 | population | 20595360 |
| Brazil | 1999 | cases | 37737 |
| Brazil | 1999 | population | 172006362 |
❌ Not Tidy! Here we fail the Tidy Data Check since cases is not a property of [Afghanistan in 1999]
![]() dplyr |
![]() ggplot2 |
![]() forcats |
![]() tibble |
![]() readr |
![]() stringr |
![]() tidyr |
![]() purrr |
readrHomepage | Overview (R4DS) | PDF Cheatsheet
Two key functions: read_csv(), write_csv() (Bonus: read_delim() if others fail)
Warning! read_csv() vs. read.csv()
Note that these are not the same as R’s built-in read.csv() and write.csv()! The built-in R functions will produce a plain data.frame object, not a tibble
read_csv() path issues either 😉)gdp_df <- read_csv("https://gist.githubusercontent.com/jpowerj/fecd437b96d0954893de727383f2eaf2/raw/fec58507f7095cb8341b229d6eb74ce53232d663/gdp_2010.csv")
gdp_df |> head(6)| name | code | year | value |
|---|---|---|---|
| Afghanistan | AFG | 2010 | 15936800636 |
| Albania | ALB | 2010 | 11926953259 |
| Algeria | DZA | 2010 | 161207268655 |
| American Samoa | ASM | 2010 | 576000000 |
| Andorra | AND | 2010 | 3355695364 |
| Angola | AGO | 2010 | 82470913121 |
tibbleHomepage | Overview (R4DS) | PDF Cheatsheet
Replaces R’s built-in data.frame, but retains syntax for backwards compatibility:
tribble() (tibble defined row-by-row)dplyrdplyr select():
arrange():
| country | year | cases | population |
|---|---|---|---|
| Afghanistan | 1999 | 745 | 19987071 |
| Afghanistan | 2000 | 2666 | 20595360 |
| Brazil | 1999 | 37737 | 172006362 |
| Brazil | 2000 | 80488 | 174504898 |
| China | 1999 | 212258 | 1272915272 |
| China | 2000 | 213766 | 1280428583 |
| country | year | cases | population |
|---|---|---|---|
| China | 2000 | 213766 | 1280428583 |
| China | 1999 | 212258 | 1272915272 |
| Brazil | 2000 | 80488 | 174504898 |
dplyr mutate():
dplyr summarize():
readr, tibble and dplyr functions help us!gdp_df <- read_csv("https://gist.githubusercontent.com/jpowerj/c83e87f61c166dea8ba7e4453f08a404/raw/29b03e6320bc3ffc9f528c2ac497a21f2d801c00/gdp_2000_2010.csv")
gdp_df |> head(5)| Country Name | Country Code | Year | Value |
|---|---|---|---|
| Afghanistan | AFG | 2010 | 15936800636 |
| Albania | ALB | 2000 | 3632043908 |
| Albania | ALB | 2010 | 11926953259 |
| Algeria | DZA | 2000 | 54790245601 |
| Algeria | DZA | 2010 | 161207268655 |
Let’s keep just the rows containing data from 2000, drop extraneous columns, and then rename the remaining columns to get rid of the spaces
gdp_2000_df <- gdp_df |>
select(`Country Name`,Year,Value) |>
filter(Year == "2000") |>
rename(country=`Country Name`, year=`Year`, gdp=`Value`)
gdp_2000_df |> write_csv("assets/gdp_2000.csv")
gdp_2000_df |> head()| country | year | gdp |
|---|---|---|
| Albania | 2000 | 3632043908 |
| Algeria | 2000 | 54790245601 |
| Andorra | 2000 | 1434429703 |
| Angola | 2000 | 9129594819 |
| Antigua and Barbuda | 2000 | 830158769 |
| Argentina | 2000 | 284203750000 |
df.merge(other_df), let’s use it 😎(see Appendix Slides → for merging via Tidyverse)
Sometimes you can’t merge because one of the datasets looks like the table on the left (non-tidy), but we need it to look like the table on the right (tidy)…
In data-cleaning jargon, this dataset is long (more than one row per observation)
In data-cleaning jargon, this dataset is wide (one row per obs; usually tidy)
pivot_wider()pivot_wider() Documentation (part of tidyr)
wide_df <- table2 |> pivot_wider(
id_cols = c(country, year),
names_from = type,
values_from = count
)
wide_df| country | year | cases | population |
|---|---|---|---|
| Afghanistan | 1999 | 745 | 19987071 |
| Afghanistan | 2000 | 2666 | 20595360 |
| Brazil | 1999 | 37737 | 172006362 |
| Brazil | 2000 | 80488 | 174504898 |
| China | 1999 | 212258 | 1272915272 |
| China | 2000 | 213766 | 1280428583 |
pivot_longer()pivot_longer() Documentation (part of tidyr)
long_df <- wide_df |> pivot_longer(
cols = c(cases, population),
names_to = "var",
values_to = "val"
)
long_df |> head()| country | year | var | val |
|---|---|---|---|
| Afghanistan | 1999 | cases | 745 |
| Afghanistan | 1999 | population | 19987071 |
| Afghanistan | 2000 | cases | 2666 |
| Afghanistan | 2000 | population | 20595360 |
| Brazil | 1999 | cases | 37737 |
| Brazil | 1999 | population | 172006362 |
forcatsreadrstringrtidyrpurrrlubridate*forcatsHomepage | Overview | Cheatsheet
R’s data structure for categorical variables)[1] "Jan" "Jan" "Feb" "Dec"
[1] Jan Jan Feb Dec
Levels: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Without forcats:
fct_infreq()Automatically combining using fct_lump():
Manually Combining using fct_recode():
gss_cat |>
mutate(partyid = fct_recode(partyid,
"Republican" = "Strong republican",
"Republican" = "Not str republican",
"Independent" = "Ind,near rep",
"Independent" = "Ind,near dem",
"Democrat" = "Not str democrat",
"Democrat" = "Strong democrat",
"Other" = "No answer",
"Other" = "Don't know",
"Other" = "Other party"
)) |>
count(partyid)| partyid | n |
|---|---|
| Other | 548 |
| Republican | 5346 |
| Independent | 8409 |
| Democrat | 7180 |
purrr: Functional ProgrammingHomepage | Overview | Cheatsheet | Tutorials, with Applications
Provides an anonymous function operator ~, arguments get named .x, .y:
lubridate*gather()2024 DSAN Bootcamp W07-2: Tidyverse