Case Law Grounding: Using Precedents to Align Decision-Making for Humans and AI

Quan Ze Chen,Amy X. Zhang
2024-09-06
Abstract:Communities and groups often need to make decisions based on social norms and preferences, such as when moderating content or building AI systems that reflect human values. The prevailing approach has been to first create high-level guidelines -- ``constitutions'' -- and then decide on new cases using the outlined criteria. However, social norms and preferences vary between groups, decision-makers can interpret guidelines inconsistently, and exceptional situations may be under-specified. In this work, we take inspiration from legal systems and introduce ``case law grounding'' (CLG), a novel workflow that uses past cases and decisions (\textbf{precedents}) to help ground future decisions, for both human and LLM-based decision-makers. We evaluate CLG against a constitution-only approach on two tasks for both types of decision-makers, and find that decisions produced with CLG were more accurately aligned to observed ground truth in all cases, producing a 3.3--23.3 \%-points improvement (across different tasks and groups) for humans and 9.2--30.0 \%-points (across different tasks and groups) for LLM agents. We also discuss other aspects where a case-based approach could augment existing ``constitutional'' approaches when it comes to aligning human and AI decisions.
Human-Computer Interaction
What problem does this paper attempt to address?