Week 6: Descriptive Statistics and Data Visualization¶
DSUA111: Data Science for Everyone, NYU, Fall 2020¶
TA Jeff, jpj251@nyu.edu¶
- This slideshow: https://jjacobs.me/dsua111-sections/week-06
- All materials: https://github.com/jpowerj/dsua111-sections
Overview¶
- Quick Pandas Recap
- Variable Types
- Frequency/Central Tendency
- Dispersion and Position
[Visualization: sprinkled throughout]
Pandas Recap¶
Where we left off...
In [1]:
import pandas as pd
gift_df = pd.read_csv("ForeignGifts_Universities.csv")
In [2]:
try:
gift_df['amount_cleaned'] = gift_df['Foreign Gift Amount'].str.replace("$","").str.replace(",","")
except Exception as e:
print(e)
In [3]:
gift_df['amount_cleaned'] = gift_df['Foreign Gift Amount'].str.replace("(","").str.replace(")","").str.replace("$","").str.replace(",","")
In [4]:
gift_df.head()
Out[4]:
In [5]:
gift_df.dtypes
Out[5]:
In [6]:
gift_df['amount_cleaned'] = gift_df['amount_cleaned'].astype(float)
In [7]:
gift_df.dtypes
Out[7]:
Variable Types?¶
Gift Amount? (the cleaned version)
In [8]:
gift_df['amount_cleaned']
Out[8]:
amount_cleaned is Numeric, Continuous
(Why not discrete?)
Number of letters in institution name?
In [9]:
gift_df['institution_length'] = gift_df['Institution Name'].apply(len)
In [10]:
gift_df.head()
Out[10]:
institution_length is Numeric, Discrete
Name of Institution?
In [11]:
gift_df['Institution Name']
Out[11]:
Institution Name is Categorical, Nominal
Rank of the gift? (1=largest, 2=second-largest, ...)
In [12]:
gift_df['gift_rank'] = gift_df['amount_cleaned'].rank(method='min', ascending=False).astype(int)
In [13]:
gift_df.head()
Out[13]:
In [14]:
gift_df.sort_values(by='gift_rank').head(5)
Out[14]:
gift_rank is Categorical, Ordinal
Measures of Frequency¶
In [15]:
gift_df['Institution Name'].value_counts().head(10)
Out[15]:
In [16]:
gift_df['Institution Name'].value_counts(normalize=True).head(10)
Out[16]:
In [17]:
gift_df['Country of Giftor'].value_counts().head(10)
Out[17]:
In [18]:
gift_df['Country of Giftor'].value_counts(normalize=True).head(10)
Out[18]:
Measures of Central Tendency¶
In [19]:
gift_df['amount_cleaned'].describe()
Out[19]:
In [20]:
gift_df['amount_cleaned'].describe().apply(lambda x: format(x, 'f'))
Out[20]:
In [21]:
import matplotlib.pyplot as plt
plt.hist(gift_df['amount_cleaned'])
plt.show()
In [22]:
plt.hist(gift_df['amount_cleaned'], bins=100)
plt.show()
In [23]:
import numpy as np
gift_df['log_amount'] = gift_df['amount_cleaned'].apply(np.log)
In [24]:
gift_df[gift_df['log_amount'] == -np.inf] = np.nan
In [25]:
plt.hist(gift_df['log_amount'], bins=100)
plt.show()
In [26]:
np.exp(13.125)
Out[26]:
and... BEWARE!¶
In [27]:
import anscombe
anscombe.plot_anscombe()
("Anscombe's Quartet": https://en.wikipedia.org/wiki/Anscombe%27s_quartet )
Measures of Dispersion¶
In [28]:
gift_df['amount_cleaned'].var()
Out[28]:
Variance in words: the average squared distance from the mean
But... why do we square it?
In [29]:
my_df = pd.DataFrame({'x':[1,2,3,4,5,6],'y':[1,2,3,-1,-2,-3]})
In [30]:
plt.scatter(my_df['x'], my_df['y']); plt.show()
In [31]:
y_mean = my_df['y'].mean(); y_mean
Out[31]:
In [32]:
my_df['y'].std()
Out[32]:
In [33]:
(my_df['y'] - y_mean).mean()
Out[33]:
And why both standard deviation and variance?¶
- Simple answer: standard deviation is in the same units as the variable.
- Standard deviation of gift amount is approx. $2.76 million
- Variance of gift amount is approx. 7.6 trillion ......?
In [34]:
%%HTML
<video width="80%" controls>
<source src="ice_cream_glove.mp4" type="video/mp4">
</video>