Human decision making balances reward maximization and policy compression
Lucy Lai,Samuel J. Gershman
DOI: https://doi.org/10.1371/journal.pcbi.1012057
2024-04-27
PLoS Computational Biology
Abstract:Policy compression is a computational framework that describes how capacity-limited agents trade reward for simpler action policies to reduce cognitive cost. In this study, we present behavioral evidence that humans prefer simpler policies, as predicted by a capacity-limited reinforcement learning model. Across a set of tasks, we find that people exploit structure in the relationships between states, actions, and rewards to "compress" their policies. In particular, compressed policies are systematically biased towards actions with high marginal probability, thereby discarding some state information. This bias is greater when there is redundancy in the reward-maximizing action policy across states, and increases with memory load. These results could not be explained qualitatively or quantitatively by models that did not make use of policy compression under a capacity limit. We also confirmed the prediction that time pressure should further reduce policy complexity and increase action bias, based on the hypothesis that actions are selected via time-dependent decoding of a compressed code. These findings contribute to a deeper understanding of how humans adapt their decision-making strategies under cognitive resource constraints. Decision making taxes cognitive resources. For example, when shopping for groceries on a budget, we must evaluate which brand offers the best value for the price. But time constraints or mental fatigue can often steer us towards familiar choices, such as sticking to the same brand. To understand how cognitive resource limitations affect human decision making, we conducted a study in which we manipulated the number of optimal choices and the time limit within which choices were made. Across three tasks, we found that people utilize task structure to compress the amount of information factored into their decision making. Information compression biases people towards their past choices. This bias persists even when multiple optimal choices are available, and intensifies under cognitive load and time pressure. A computational model of decision making under cognitive constraints accurately describes the experimental data. Our findings may have the potential to inform the design of choice environments that better align with human decision biases.
biochemical research methods,mathematical & computational biology