Abstract:Abstract People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.

How Humans Perceive Human-like Behavior in Video Game Navigation

Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games

Inferring Human Vision in a Human-Like Way: Key Factors Influencing the Cognitive Processing of Level-1 Visual Perspective-Taking

Human-AI Collaboration in a Cooperative Game Setting

Eyes on the Game: Deciphering Implicit Human Signals to Infer Human Proficiency, Trust, and Intent

Generative Personas That Behave and Experience Like Humans

Human or Not? A Gamified Approach to the Turing Test

Human‐Like Interactive Behavior Generation for Autonomous Vehicles: A Bayesian Game‐Theoretic Approach with Turing Test

How is the AI Perceived when It Behaves (Un)fairly?

Toward Human-AI Alignment in Large-Scale Multi-Player Games

Can Machines Imitate Humans? Integrative Turing Tests for Vision and Language Demonstrate a Narrowing Gap

React to This! How Humans Challenge Interactive Agents using Nonverbal Behaviors

Towards personalized human AI interaction - adapting the behavior of AI agents using neural signatures of subjective interest

Human Learning about AI Performance

Immersive machine learning for social attitude detection in virtual reality narrative games

Perception of Personality Traits in Crowds of Virtual Humans

Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction

How Can I Assist You Today?: A Comparative Analysis of a Humanoid Robot and a Virtual Human Avatar in Human Perception

Improving Humanness of Virtual Agents and Users' Cooperation through Emotions

ImCasting: Nonverbal Behaviour Reinforcement Learning of Virtual Humans through Adaptive Immersive Game

Navigational Behavior of Humans and Deep Reinforcement Learning Agents