Abstract:A broad current application of algorithms is in formal and quantitative measures of murky concepts -- like merit -- to make decisions. When people strategically respond to these sorts of evaluations in order to gain favorable decision outcomes, their behavior can be subjected to moral judgments. They may be described as 'gaming the system' or 'cheating,' or (in other cases) investing 'honest effort' or 'improving.' Machine learning literature on strategic behavior has tried to describe these dynamics by emphasizing the efforts expended by decision subjects hoping to obtain a more favorable assessment -- some works offer ways to preempt or prevent such manipulations, some differentiate 'gaming' from 'improvement' behavior, while others aim to measure the effort burden or disparate effects of classification systems. We begin from a different starting point: that the design of an evaluation itself can be understood as furthering goals held by the evaluator which may be misaligned with broader societal goals. To develop the idea that evaluation represents a strategic interaction in which both the evaluator and the subject of their evaluation are operating out of self-interest, we put forward a model that represents the process of evaluation using three interacting agents: a decision subject, an evaluator, and society, representing a bundle of values and oversight mechanisms. We highlight our model's applicability to a number of social systems where one or two players strategically undermine the others' interests to advance their own. Treating evaluators as themselves strategic allows us to re-cast the scrutiny directed at decision subjects, towards the incentives that underpin institutional designs of evaluations. The moral standing of strategic behaviors often depend on the moral standing of the evaluations and incentives that provoke such behaviors.

When Performance Trumps Gender Bias: Joint vs. Separate Evaluation

Inside the Black Box of Organizational Life: The Gendered Language of Performance Assessment

A meta-analysis of gender stereotypes and bias in experimental simulations of employment decision making.

Paternalism in the Performance Context: Evaluators Who Feel Social Pressure to Avoid Exhibiting Prejudice Deliver More Inflated Performance Feedback to Women

Gender and cultural bias in student evaluations: Why representation matters

Tokenism in performance evaluation: The effects of work group representation on male-female and White-Black differences in performance ratings.

Discovery of Bias and Strategic Behavior in Crowdsourced Performance Assessment

Overcoming the Outcome Bias: Making Process Matter

What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring

Interventions That Affect Gender Bias in Hiring: A Systematic Review

The gender equity gap: A multistudy investigation of within-job inequality in equity-based awards.

Efficient candidate screening under multiple tests and implications for fairness

Discrimination and Constraints: Evidence from The Voice

Gender equity in hiring: examining the effectiveness of a personality-based algorithm

Biased Humans, (Un)Biased Algorithms?

Modeling gender differences in the job promotion process: Replication and extension of Martell et al. (1996).

Strategic Evaluation: Subjects, Evaluators, and Society

Sex, age, and equity behavior.

Gender bias in teaching evaluations: the causal role of department gender composition

Overcoming the Outcome Bias: Making Intentions Matter

Choose Your Lenses: Flaws in Gender Bias Evaluation