Abstract:Lifting an object requires precise scaling of fingertip forces based on a prediction of object weight. At object contact, a series of tactile and visual events arise that need to be rapidly processed online to fine-tune the planned motor commands for lifting the object. The brain mechanisms underlying multisensory integration serially at transient sensorimotor events, a general feature of actions requiring hand-object interactions, are not yet understood. In this study we tested the relative weighting between haptic and visual signals when they are integrated online into the motor command. We used a new virtual reality setup to desynchronize visual feedback from haptics, which allowed us to probe the relative contribution of haptics and vision in driving participants’ movements when they grasped virtual objects simulated by two force-feedback robots. We found that visual delay changed the profile of fingertip force generation and led participants to perceive objects as heavier than when lifts were performed without visual delay. We further modeled the effect of vision on motor output by manipulating the extent to which delayed visual events could bias the force profile, which allowed us to determine the specific weighting the brain assigns to haptics and vision. Our results show for the first time how visuo-haptic integration is processed at discrete sensorimotor events for controlling object-lifting dynamics and further highlight the organization of multisensory signals online for controlling action and perception. NEW & NOTEWORTHY Dexterous hand movements require rapid integration of information from different senses, in particular touch and vision, at different key time points as movement unfolds. The relative weighting between vision and haptics for object manipulation is unknown. We used object lifting in virtual reality to desynchronize visual and haptic feedback and find out their relative weightings. Our findings shed light on how rapid multisensory integration is processed over a series of discrete sensorimotor control points.

Absence of Inertial Load on Hand Decreases Task Performance in Virtual Reality Interaction

The Virtual Hand Illusion is Moderated by Context-Induced Spatial Reference Frames.

Enhancing weight perception in virtual reality: an analysis of kinematic features

Age-related Differences in Upper Limb Motor Performance and Intrinsic Motivation During a Virtual Reality Task

Arm Posture Changes and Influences on Hand Controller Interaction Evaluation in Virtual Reality

Eye movement characteristics and visual fatigue assessment of virtual reality games with different interaction modes

Physical load reduces synchrony effects on agency and ownership in the virtual hand illusion

Examining the Fine Motor Control Ability of Linear Hand Movement in Virtual Reality.

An Empirical Evaluation on Arm Fatigue in Free Hand Interaction and Guidelines for Designing Natural User Interfaces in VR.

Reducing Arm Fatigue in Virtual Reality by Introducing 3D-Spatial Offset

Hand-adaptive User Interface: Improved Gestural Interaction in Virtual Reality

Comparing eye–hand coordination between controller-mediated virtual reality, and a real-world object interaction task

The Effect of Audiovisual Spatial Design on User Experience of Bare-Hand Interaction in VR

Visual delay affects force scaling and weight perception during object lifting in virtual reality

Effect of Physical and Virtual Feedback on Reach-to-Grasp Movements in Virtual Environments

Innovative Weight Simulation in Virtual Reality Cube Games: A Pseudo-Haptic Approach

Performance And User Preference Of Various Functions For Mapping Hand Position To Movement Velocity In A Virtual Environment

Effects of Transfer Functions and Body Parts on Body-Centric Locomotion in Virtual Reality

Evaluating the Effects of Real World Distraction on User Performance in Virtual Environments

Designing and Analyzing In-Place Motor Tasks in Virtual Reality With Goal Functions

Alterations in Physical Demands During Virtual/Augmented Reality-Based Tasks: A Systematic Review