Abstract:How can we train an assistive human-machine interface (e.g., an electromyography-based limb prosthesis) to translate a user's raw command signals into the actions of a robot or computer when there is no prior mapping, we cannot ask the user for supervision in the form of action labels or reward feedback, and we do not have prior knowledge of the tasks the user is trying to accomplish? The key idea in this paper is that, regardless of the task, when an interface is more intuitive, the user's commands are less noisy. We formalize this idea as a completely unsupervised objective for optimizing interfaces: the mutual information between the user's command signals and the induced state transitions in the environment. To evaluate whether this mutual information score can distinguish between effective and ineffective interfaces, we conduct an observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in a variety of domains, with an average Spearman's rank correlation of 0.43. In addition to offline evaluation of existing interfaces, we use our unsupervised objective to learn an interface from scratch: we randomly initialize the interface, have the user attempt to perform their desired tasks using the interface, measure the mutual information score, and update the interface to maximize mutual information through reinforcement learning. We evaluate our method through a user study with 12 participants who perform a 2D cursor control task using a perturbed mouse, and an experiment with one user playing the Lunar Lander game using hand gestures. The results show that we can learn an interface from scratch, without any user supervision or prior knowledge of tasks, in under 30 minutes.

A Bayesian Experimental Design Approach Maximizing Information Gain for Human-Computer Interaction

BIGnav: Bayesian Information Gain for Guiding Multiscale Navigation.

Bayesian Information Gain to Design Interaction

CO-BED: Information-Theoretic Contextual Optimization via Bayesian Experimental Design

Expert-guided Bayesian Optimisation for Human-in-the-loop Experimental Design of Known Systems

Optimal design of experiments to identify latent behavioral types

Sequential Bayesian experimental designs via reinforcement learning

CARE: Confidence-rich Autonomous Robot Exploration Using Bayesian Kernel Inference and Optimization

Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvement

Human-Algorithm Collaborative Bayesian Optimization for Engineering Systems

Bayesian Optimal Design of Experiments For Inferring The Statistical Expectation Of A Black-Box Function

An Induced Natural Selection Heuristic for Finding Optimal Bayesian Experimental Designs

Information-Theoretic Analysis Of Human Performance For Command Selection

First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization

Bayesian Experimental Design via Contrastive Diffusions

A Bayesian Shared Control Approach for Wheelchair Robot with Brain Machine Interface

LIMIT: Learning Interfaces to Maximize Information Transfer

Exploration Of New-Generation Human Computer Interface Based On Participatory Design Strategy

Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties

Bayesian Optimal Design of Experiments for Inferring the Statistical Expectation of Expensive Black-Box Functions