Abstract:As Generative AI rises in adoption, its use has expanded to include domains such as hiring and recruiting. However, without examining the potential of bias, this may negatively impact marginalized populations, including people with disabilities. To address this important concern, we present a resume audit study, in which we ask ChatGPT (specifically, GPT-4) to rank a resume against the same resume enhanced with an additional leadership award, scholarship, panel presentation, and membership that are disability related. We find that GPT-4 exhibits prejudice towards these enhanced CVs. Further, we show that this prejudice can be quantifiably reduced by training a custom GPTs on principles of DEI and disability justice. Our study also includes a unique qualitative analysis of the types of direct and indirect ableism GPT-4 uses to justify its biased decisions and suggest directions for additional bias mitigation work. Additionally, since these justifications are presumably drawn from training data containing real-world biased statements made by humans, our analysis suggests additional avenues for understanding and addressing human bias.
What problem does this paper attempt to address?
### What problems does this paper attempt to solve?
This paper aims to solve the problem of bias against disabled people in the resume screening process by generative AIs (such as ChatGPT). Specifically, through a resume audit study, the author explored the following key issues:
1. **Existence of disability bias**: Will GPT - 4 show bias towards resumes containing disability - related information? Will this bias vary depending on the type of disability?
- **Research Question 1 (RQ1)**: Does GPT - 4 have bias against resumes that mention disability when screening resumes? Will different types of disability lead to different degrees of bias?
2. **Methods to reduce bias**: By training a customized GPT model (based on the principles of diversity and inclusion and disability justice), can this bias be effectively reduced?
- **Research Question 2 (RQ2)**: Does a GPT trained with the principles of diversity, equity, and inclusion (DEI) show less bias than a general - purpose GPT?
3. **Analysis of the causes of bias**: When GPT explains its ranking decisions, does it provide evidence that can reveal the source or type of potential bias?
- **Research Question 3 (RQ3)**: Can direct or indirect discriminatory reasoning be found in GPT's explanations for resume rankings?
### Research background and motivation
With the increasing application of generative artificial intelligence (Generative AI, GAI) in recruitment and human resource management, using AI for resume screening has become a common practice. However, these systems may inadvertently amplify existing social biases, especially against marginalized groups (such as disabled people). Although AI can optimize the resume screening process and save time, if not carefully scrutinized, it may exacerbate inequality in the job market.
Previous studies have shown that AI systems may show bias when processing resumes involving characteristics such as gender and age. However, the bias of AI systems in resume screening involving disabled people has not been fully quantified and studied. Therefore, it is crucial to understand and evaluate the bias of GAI in this area to ensure that AI tools do not further impede the opportunities of disabled people in the job market.
### Overview of methods
To evaluate the bias of GPT - 4 against disabled people in resume screening, the researchers designed a mixed - method resume audit study. They created a standard resume (Control CV) and six enhanced resumes (Enhanced Curricula Vitae, ECV), each ECV containing information related to different types of disabilities. The study was carried out through the following steps:
1. **Resume creation**: Using a public resume as a basis, create multiple versions of the resume, some of which contain awards, scholarships, speeches, and organizational memberships related to specific disabilities.
2. **AI selection and preparation**: Two versions of GPT - 4 were selected for testing: one was the unmodified standard GPT - 4, and the other was a custom - made GPT (Disability - Aware GPT, DA - GPT) specifically trained to reduce disability bias.
3. **Data collection**: Let these two versions of GPT rank each pair of resumes (control group and enhanced group) multiple times respectively, and record the results and explanations of each ranking.
4. **Data analysis**: Through quantitative and qualitative analysis, evaluate the degree of bias and its changes under different conditions.
### Main findings
- **Quantitative analysis**: The results show that GPT - 4 does have bias against resumes containing disability information, and this bias varies among different types of disabilities. The DA - GPT trained with DEI principles significantly reduces this bias.
- **Qualitative analysis**: Through text analysis of GPT's explanations, a variety of direct and indirect discriminatory reasoning were found, such as misinterpreting disability disclosure as part of diversity work, or believing that candidates are distracted and have a narrow research scope.
### Conclusions and recommendations
This study emphasizes the importance of identifying and reducing disability bias in AI recruitment tools. Future work should continue to explore how to improve AI systems to make them more fair and inclusive, thereby promoting equal opportunities for disabled people in the job market.