Abstract:In the rapidly evolving field of artificial intelligence, large language models (LLMs) have emerged as powerful tools for a myriad of applications, from natural language processing to decision-making support systems. However, as these models become increasingly integrated into societal frameworks, the imperative to ensure they operate within ethical and moral boundaries has never been more critical. This paper introduces a novel benchmark designed to measure and compare the moral reasoning capabilities of LLMs. We present the first comprehensive dataset specifically curated to probe the moral dimensions of LLM outputs, addressing a wide range of ethical dilemmas and scenarios reflective of real-world complexities. The main contribution of this work lies in the development of benchmark datasets and metrics for assessing the moral identity of LLMs, which accounts for nuance, contextual sensitivity, and alignment with human ethical standards. Our methodology involves a multi-faceted approach, combining quantitative analysis with qualitative insights from ethics scholars to ensure a thorough evaluation of model performance. By applying our benchmark across several leading LLMs, we uncover significant variations in moral reasoning capabilities of different models. These findings highlight the importance of considering moral reasoning in the development and evaluation of LLMs, as well as the need for ongoing research to address the biases and limitations uncovered in our study. We publicly release the benchmark at <a class="link-external link-https" href="https://drive.google.com/drive/u/0/folders/1k93YZJserYc2CkqP8d4B3M3sgd3kA8W7" rel="external noopener nofollow">this https URL</a> and also open-source the code of the project at <a class="link-external link-https" href="https://github.com/agiresearch/MoralBench" rel="external noopener nofollow">this https URL</a>.

Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs

Alignment Between the Decision-Making Logic of LLMs and Human Cognition: A Case Study on Legal LLMs

Exploring and steering the moral compass of Large Language Models

Language Model Alignment in Multilingual Trolley Problems

Exploring the psychology of LLMs' Moral and Legal Reasoning

Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation

Evaluating Moral Beliefs across LLMs through a Pluralistic Framework

Large-scale moral machine experiment on large language models

When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

Evaluating the Moral Beliefs Encoded in LLMs

MoralBench: Moral Evaluation of LLMs

Argumentative Large Language Models for Explainable and Contestable Decision-Making

Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs

Inducing Human-like Biases in Moral Reasoning Language Models

Moral Alignment for LLM Agents

Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation

Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in

The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making

Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experiment