Abstract:Immersive software tools are virtual environments designed to give their users an augmented view of real-world data and ways of manipulating that data. As virtual environments, every action users make while interacting with these tools can be carefully logged, as can the state of the software and the information it presents to the user, giving these actions context. This data provides a high-resolution lens through which dynamic cognitive and behavioral processes can be viewed. In this report, we describe new methods for the analysis and interpretation of such data, utilizing a novel implementation of the Beta Process Hidden Markov Model (BP-HMM) for analysis of software activity logs. We further report the results of a preliminary study designed to establish the validity of our modeling approach. A group of 20 participants were asked to play a simple computer game, instrumented to log every interaction with the interface. Participants had no previous experience with the game's functionality or rules, so the activity logs collected during their naïve interactions capture patterns of exploratory behavior and skill acquisition as they attempted to learn the rules of the game. Pre- and post-task questionnaires probed for self-reported styles of problem solving, as well as task engagement, difficulty, and workload. We jointly modeled the activity log sequences collected from all participants using the BP-HMM approach, identifying a global library of activity patterns representative of the collective behavior of all the participants. Analyses show systematic relationships between both pre- and post-task questionnaires, self-reported approaches to analytic problem solving, and metrics extracted from the BP-HMM decomposition. Overall, we find that this novel approach to decomposing unstructured behavioral data within software environments provides a sensible means for understanding how users learn to integrate software functionality for strategic task pursuit.

Using brain imaging to track problem solving in a complex state space

Using Brain Imaging to Interpret Student Problem Solving

Characterizing and Differentiating Brain State Dynamics Via Hidden Markov Models

Modeling Brain Functional Dynamics Via Hidden Markov Models

Exploring Latent States of Problem-Solving Competence Using Hidden Markov Model on Process Data

Simulating Human Heuristic Problem Solving: A Study By Combining Act-R And Fmri Brain Image

Using fMRI to Test Models of Complex Cognition.

Using Svm to Predict High-Level Cognition from Fmri Data: A Case Study of 4*4 Sudoku Solving

Modeling strategic use of human computer interfaces with novel hidden Markov models

A Computational Cognitive Model for Simulating Heuristics Retrieval in Human Problem Solving

Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

A Realistic Framework for Investigating Decision Making in the Brain With High Spatiotemporal Resolution Using Simultaneous EEG/fMRI and Joint ICA

Bidirectional understanding and cooperation: interbrain neural synchronization during social navigation

Brain state and dynamic transition patterns of motor imagery revealed by the bayes hidden markov model

Using Brain Imaging to Extract the Structure of Complex Events at the Rational Time Band

Forming cognitive maps for abstract spaces: the roles of the human hippocampus and orbitofrontal cortex

Brain Mechanisms of Intuitive Problem Solving in Board Game Experts

Discovering Optimal Brain States For Problem Solving With Eeg

MIND-Map; A Comprehensive Toolbox for Estimating Brain Dynamic States

Characterizing functional hippocampal pathways in a brain-based device as it solves a spatial memory task

Bridging Cognitive Maps: a Hierarchical Active Inference Model of Spatial Alternation Tasks and the Hippocampal-Prefrontal Circuit