Abstract:Increasing interest in ensuring safety of next-generation Artificial Intelligence (AI) systems calls for novel approaches to embedding morality into autonomous agents. Traditionally, this has been done by imposing explicit top-down rules or hard constraints on systems, for example by filtering system outputs through pre-defined ethical rules. Recently, instead, entirely bottom-up methods for learning implicit preferences from human behavior have become increasingly popular, such as those for training and fine-tuning Large Language Models. In this paper, we provide a systematization of existing approaches to the problem of introducing morality in machines - modeled as a continuum, and argue that the majority of popular techniques lie at the extremes - either being fully hard-coded, or entirely learned, where no explicit statement of any moral principle is required. Given the relative strengths and weaknesses of each type of methodology, we argue that more hybrid solutions are needed to create adaptable and robust, yet more controllable and interpretable agents. In particular, we present three case studies of recent works which use learning from experience (i.e., Reinforcement Learning) to explicitly provide moral principles to learning agents - either as intrinsic rewards, moral logical constraints or textual principles for language models. For example, using intrinsic rewards in Social Dilemma games, we demonstrate how it is possible to represent classical moral frameworks for agents. We also present an overview of the existing work in this area in order to provide empirical evidence for the potential of this hybrid approach. We then discuss strategies for evaluating the effectiveness of moral learning agents. Finally, we present open research questions and implications for the future of AI safety and ethics which are emerging from this framework.

Moral Association Graph: A Cognitive Model for Automated Moral Inference

Towards Human-like Perception: Learning Structural Causal Model in Heterogeneous Graph

Contextualized moral inference

Moral Sparks in Social Media Narratives

A Computational Model of Commonsense Moral Decision Making

Using Machine Learning to Guide Cognitive Modeling: A Case Study in Moral Reasoning

Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences

Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theory

When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment

Attributions toward Artificial Agents in a modified Moral Turing Test

MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks

Moral dynamics: Grounding moral judgment in intuitive physics and intuitive psychology

The Puzzle of Evaluating Moral Cognition in Artificial Agents

Human-centred artificial intelligence: a contextual morality perspective

COKE: A Cognitive Knowledge Graph for Machine Theory of Mind

Neurosymbolic AI for Reasoning on Graph Structures: A Survey

Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity

Cognitive Models as Simulators: The Case of Moral Decision-Making

Learning Machine Morality through Experience and Interaction

Learning Moral Graphs in Construction of High-Dimensional Bayesian Networks for Mixed Data