Week 10: Econometric Policy Evaluation and Inverse Fairness

DSAN 5450: Data Ethics and Policy
Spring 2024, Georgetown University

Author

Affiliation

Jeff Jacobs

jj1088@georgetown.edu

Open slides in new window →

Policy Evaluation in General

Why Call It “Econometric” Policy Evaluation?

• In the same way that data science can be used for good or for not-so-good…
• Economics provides methods that allow us to evaluate a policy with respect to an ethical framework
• Typically efficiency and/or profit maximization, but just as easily Rawlsian or Roemerian
• “The master’s tools will never dismantle the master’s house”❓❓❓
• (“It’s certainly true that the master uses the tool of [data science], but that doesn’t mean he owns it.”)

The Tools of Economics as Wielded by Economists

• Fun in-class pop quiz!

The Tools of Economics as Wielded by Economists

• Did you answer correctly???

The Tools of Economics as Wielded by Economists

• Addendum / reminder (see also)

EAAMO 2023 Accepted Papers

• Equity and Access in Algorithms, Mechanisms, and Optimization

• “Disincentivizing Polarization in Social Networks”

• “30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity”

  Assignments graded later in the sequence tend to (1) receive lower grades, (2) receive comments that are notably more negative and less polite, and (3) exhibit lower grading quality measured by post-grade complaints.

• “Potty Parity: Process Flexibility via Unisex Restroom”

  Having a one-unit unisex restroom, either by converting a unit of the men’s restroom or building an additional one, goes a long way toward improving wait time or user utility, and reducing their disparities

• “Overbooking with Priority-Respecting Reassignment”

Co-Founder of MD4SG \(\rightarrow\) EAAMO: Rediet Abebe

From “When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks”, EMNLP 2023

Lily Hu

• CS excludes ethical considerations “by design”, via focus on how computers do work, not how they ought to work.
• The latter “falls within the purview of the normative disciplines, most notably ethics, a subfield of philosophy. For students to succeed at learning not only how to build innovative computing systems, but also how to determine whether they should build those systems or how ethical considerations should constrain their design, it is imperative that these two disciplines work together. To this end, Harvard CS and Philosophy faculty have been partnering to develop the Embedded EthiCS curriculum.”

Embedded EthiCS

Ninja Coder Hackerz / Think It! Dream It! Do It!

• CS: Learn The Coolest, Newest Stuff, Move Fast And Break Things!!!
• Ethics: Listen to a man in a tweed jacket with elbow pads pontificate about Kant for 3 hours
• The TED Talk Circuit

Social Welfare Functionals

Functionals?

• You probably know what a function \(f(x)\) is; a functional is a function of functions: \(\mathscr{G}(f)\)
• It’s from math, which is scary, but it’s just notation to remind us that we’re analyzing functions of functions
• In our case, they “work the same way” as regular functions, e.g., \(\mathscr{G}(f,g) = f^2 + g^2\), so \(f(x) = x, g(x) = 2x \Rightarrow \mathscr{G}(f,g)(x) = x^2 + 4x^2 = 5x^2\)

We Live In A Society, Part 2

• Utilitarianism, Kantian Ethics, and Rawls can all be modeled as Social Welfare Functionals

\[ W(\mathbf{u}) = W(u_1, \ldots, u_n) \Rightarrow W(\mathbf{u})(x) = W(u_1(x), \ldots, u_n(x)) \]

• \(u_i(x)\): Given bundle of resources \(x\), how much utility does \(i\) experience? \(u_i: \mathcal{X} \rightarrow \mathbb{R}\)
• \(W(\mathbf{u})\): Aggregates \(u_i(x)\) over all \(i\), to produce measure of overall welfare of society. \(W: (\mathcal{X} \rightarrow \mathbb{R})^N \rightarrow \mathbb{R}\).
• \(W(\mathbf{u}) = \sum_{i=1}^n \omega_iu_i(x)\). \(\omega_i\) is \(i\)’s welfare weight
• (Preview) Decomposition: evaluate policies by estimating marginal utility \(u'_i(x)\) compared to \(\omega_i\))

Alternative SWF Specifications

• Social values

\[ W(\underbrace{v_1, \ldots, v_n}_{\text{Values}})(x) \overset{\text{e.g.}}{=} \omega_1\underbrace{v_1(x)}_{\text{Privacy}} + \omega_2\underbrace{v_2(x)}_{\mathclap{\text{Public Health}}} \]

• Stakeholder Analysis

\[ W(\underbrace{s_1, \ldots, s_n}_{\text{Stakeholders}})(x) = \omega_1\underbrace{u_{s_1}(x)}_{\text{Teachers}} + \omega_2\underbrace{u_{s_2}(x)}_{\text{Parents}} + \omega_3\underbrace{u_{s_3}(x)}_{\text{Students}} + \omega_4\underbrace{u_{s_4}(x)}_{\mathclap{\text{Community}}} \]

• (Adapted from this great intro video!)

Utilitarian SWF

• Easy mode (possibly/probably your intuition?): Everyone’s welfare weight should be equal, \(\omega_i = \frac{1}{n}\)

\[ W(u_1, \ldots, u_n)(x) = \frac{1}{n}u_1(x) + \cdots + \frac{1}{n}u_n(x) \]

• \(\implies\) Utilitarian Social Welfare Functional!
• The Silly Problem of Utilitarian SWF: What if everyone is made happy by \(u_{\text{Jeef}} = -\infty\)?

The Hard Problem of Utilitarian SWF

While the rhetoric of “all men [sic] are born equal” is typically taken to be part and parcel of egalitarianism, the effect of ignoring the interpersonal variations can, in fact, be deeply inegalitarian, in hiding the fact that equal consideration for all may demand very unequal treatment in favour of the disadvantaged (Sen 1992)

• \(\implies\) “Equality of What?”
• What is the “thing” that egalitarianism obligates us to equalize (the equilisandum/equilisanda): Utility? Opportunity? Resources? Money? Freedom?

Policy Evaluation via Inverse Fairness

We Can Finally Understand This Image from Week 1!

References

Sen, Amartya. 1992. Inequality Reexamined. Clarendon Press.