Week 10: Econometric Policy Evaluation and Inverse Fairness

DSAN 5450: Data Ethics and Policy
Spring 2025, Georgetown University

Jeff Jacobs

jj1088@georgetown.edu

Wednesday, March 26, 2025

Policy Evaluation in General

Why Call It “Econometric” Policy Evaluation?

  • In the same way that data science can be used for good or for not-so-good…
  • Economics provides methods that allow us to evaluate a policy with respect to an ethical framework
  • Typically efficiency and/or profit maximization, but just as easily Rawlsian or Roemerian
  • “The master’s tools will never dismantle the master’s house”❓❓❓
  • (“It’s certainly true that the master uses the tool of [data science], but that doesn’t mean he owns it.”)

The Tools of Economics as Wielded by Economists

  • Fun in-class pop quiz!

The Tools of Economics as Wielded by Economists

  • Did you answer correctly???

The Tools of Economics as Wielded by Economists

EAAMO 2023 Accepted Papers

Co-Founder of MD4SG \(\rightarrow\) EAAMO: Rediet Abebe

From “When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks”, EMNLP 2023

Lily Hu

  • CS excludes ethical considerations “by design”, via focus on how computers do work, not how they ought to work.
  • The latter “falls within the purview of the normative disciplines, most notably ethics, a subfield of philosophy. For students to succeed at learning not only how to build innovative computing systems, but also how to determine whether they should build those systems or how ethical considerations should constrain their design, it is imperative that these two disciplines work together. To this end, Harvard CS and Philosophy faculty have been partnering to develop the Embedded EthiCS curriculum.”

Ninja Coder Hackerz / Think It! Dream It! Do It!

  • CS: Learn The Coolest, Newest Stuff, Move Fast And Break Things!!!
  • Ethics: Listen to a man in a tweed jacket with elbow pads pontificate about Kant for 3 hours
  • The TED Talk Circuit

Social Welfare Functionals

Functionals?

  • You probably know what a function \(f(x)\) is; a functional is a function of functions: \(\mathscr{G}(f)\)
  • It’s from math, which is scary, but it’s just notation to remind us that we’re analyzing functions of functions
  • In our case, they “work the same way” as regular functions, e.g., \(\mathscr{G}(f,g) = f^2 + g^2\), so \(f(x) = x, g(x) = 2x \Rightarrow \mathscr{G}(f,g)(x) = x^2 + 4x^2 = 5x^2\)

We Live In A Society, Part 2

  • Utilitarianism, Kantian Ethics, and Rawls can all be modeled as Social Welfare Functionals

\[ W(\mathbf{u}) = W(u_1, \ldots, u_n) \Rightarrow W(\mathbf{u})(x) = W(u_1(x), \ldots, u_n(x)) \]

  • \(u_i(x)\): Given bundle of resources \(x\), how much utility does \(i\) experience? \(u_i: \mathcal{X} \rightarrow \mathbb{R}\)
  • \(W(\mathbf{u})\): Aggregates \(u_i(x)\) over all \(i\), to produce measure of overall welfare of society. \(W: (\mathcal{X} \rightarrow \mathbb{R})^N \rightarrow \mathbb{R}\).
  • \(W(\mathbf{u}) = \sum_{i=1}^n \omega_iu_i(x)\). \(\omega_i\) is \(i\)’s welfare weight
  • (Preview) Decomposition: evaluate policies by estimating marginal utility \(u'_i(x)\) compared to \(\omega_i\))

Alternative SWF Specifications

  • Social values

\[ W(\underbrace{v_1, \ldots, v_n}_{\text{Values}})(x) \overset{\text{e.g.}}{=} \omega_1\underbrace{v_1(x)}_{\text{Privacy}} + \omega_2\underbrace{v_2(x)}_{\mathclap{\text{Public Health}}} \]

  • Stakeholder Analysis

\[ W(\underbrace{s_1, \ldots, s_n}_{\text{Stakeholders}})(x) = \omega_1\underbrace{u_{s_1}(x)}_{\text{Teachers}} + \omega_2\underbrace{u_{s_2}(x)}_{\text{Parents}} + \omega_3\underbrace{u_{s_3}(x)}_{\text{Students}} + \omega_4\underbrace{u_{s_4}(x)}_{\mathclap{\text{Community}}} \]

Utilitarian SWF

  • Easy mode (possibly/probably your intuition?): Everyone’s welfare weight should be equal, \(\omega_i = \frac{1}{n}\)

\[ W(u_1, \ldots, u_n)(x) = \frac{1}{n}u_1(x) + \cdots + \frac{1}{n}u_n(x) \]

  • \(\implies\) Utilitarian Social Welfare Functional!
  • The Silly Problem of Utilitarian SWF: What if everyone is made happy by \(u_{\text{Jeef}} = -\infty\)?

The Hard Problem of Utilitarian SWF

While the rhetoric of “all men [sic] are born equal” is typically taken to be part and parcel of egalitarianism, the effect of ignoring the interpersonal variations can, in fact, be deeply inegalitarian, in hiding the fact that equal consideration for all may demand very unequal treatment in favour of the disadvantaged (Sen 1992)

  • \(\implies\) “Equality of What?”
  • What is the “thing” that egalitarianism obligates us to equalize (the equilisandum/equilisanda): Utility? Opportunity? Resources? Money? Freedom?

Policy Evaluation via Inverse Fairness

We Can Finally Understand This Image from Week 1!

References

Sen, Amartya. 1992. Inequality Reexamined. Clarendon Press.

DSAN 5450 Week 10: Econometric Policy Evaluation, Inverse Fairness