Abstract:An increasing number of studies use gender information to understand phenomena such as gender bias, inequity in access and participation, or the impact of the Covid pandemic response. Unfortunately, most datasets do not include self-reported gender information, making it necessary for researchers to infer gender from other information, such as names or names and country information. An important limitation of these tools is that they fail to appropriately capture the fact that gender exists on a non-binary scale, however, it remains important to evaluate and compare how well these tools perform in a variety of contexts. In this paper, we compare the performance of a generative Artificial Intelligence (AI) tool ChatGPT with three commercially available list-based and machine learning-based gender inference tools (Namsor, Gender-API, and <a class="link-external link-http" href="http://genderize.io" rel="external noopener nofollow">this http URL</a>) on a unique dataset. Specifically, we use a large Olympic athlete dataset and report how variations in the input (e.g., first name and first and last name, with and without country information) impact the accuracy of their predictions. We report results for the full set, as well as for the subsets: medal versus non-medal winners, athletes from the largest English-speaking countries, and athletes from East Asia. On these sets, we find that Namsor is the best traditional commercially available tool. However, ChatGPT performs at least as well as Namsor and often outperforms it, especially for the female sample when country and/or last name information is available. All tools perform better on medalists versus non-medalists and on names from English-speaking countries. Although not designed for this purpose, ChatGPT may be a cost-effective tool for gender prediction. In the future, it might even be possible for ChatGPT or other large scale language models to better identify self-reported gender rather than report gender on a binary scale.

Interactive Gender Inference with Integer Linear Programming

Interactive Gender Inference in Social Media

Evaluating Gender Bias in Natural Language Inference

The Integer Linear Programming Inference Cookbook

Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation

A Graph-Based Approach for Relating Integer Programs

Inferring gender from name: a large scale performance evaluation study

Automatically Inferring Gender Associations from Language

Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models

Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints

Leveraging Interactive Knowledge And Unlabeled Data In Gender Classification With Co-Training

Gender Bias in Large Language Models across Multiple Languages

Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting

Gender inference: can chatGPT outperform common commercial tools?

Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias

Measuring Gender and Racial Biases in Large Language Models

Uncovering Implicit Gender Bias in Narratives through Commonsense Inference

Multi-Dimensional Gender Bias Classification

A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference

Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels

An interactive approach based on a discrete differential evolution algorithm for a class of integer bilevel programming problems