Social Choice for AI Alignment: Dealing with Diverse Human Feedback
William S. Zwicker,Rachel Freedman,Milan Mossé,J. Heitzig,Stuart Russell,Wesley H. Holliday,Hailey Schoelkopf,Vincent Conitzer,Emanuel Tewolde,Eric Pacuit,Nathan Lambert,Bob M. Jacobs
DOI: https://doi.org/10.48550/arXiv.2404.10271
Abstract:Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, so that, for example, they refuse to comply with requests for help with committing crimes or with producing racist text. One approach to fine-tuning, called reinforcement learning from human feed-back , learns from humans’ expressed preferences over multiple outputs. Another approach is constitutional AI , in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about “collective” preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent work-shop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023.
Computer Science,Philosophy