Abstract:Some of the barriers preventing virtual reality (VR) from being widely adopted are the cost and unfamiliarity of VR systems. Here, we propose that in many cases, the specialized controllers shipped with most VR head-mounted displays can be replaced by a regular smartphone, cutting the cost of the system, and allowing users to interact in VR using a device they are already familiar with. To achieve this, we developed SmartVR Pointer, an approach that uses smartphones to replace the specialized controllers for two essential operations in VR: selection and navigation by teleporting. In SmartVR Pointer, a camera mounted on the head-mounted display (HMD) is tilted downwards so that it points to where the user will naturally be holding their phone in front of them. SmartVR Pointer supports three selection modalities: tracker based, gaze based, and combined/hybrid. In the tracker-based SmartVR Pointer selection, we use image-based tracking to track a QR code displayed on the phone screen and then map the phone's position to a pointer shown within the field of view of the camera in the virtual environment. In the gaze-based selection modality, the user controls the pointer using their gaze and taps on the phone for selection. The combined technique is a hybrid between gaze-based interaction in VR and tracker-based Augmented Reality. It allows the user to control a VR pointer that looks and behaves like a mouse pointer by moving their smartphone to select objects within the virtual environment, and to interact with the selected objects using the smartphone's touch screen. The touchscreen is used for selection and dragging. The SmartVR Pointer is simple and requires no calibration and no complex hardware assembly or disassembly. We demonstrate successful interactive applications of SmartVR Pointer in a VR environment with a demo where the user navigates in the virtual environment using teleportation points on the floor and then solves a Tetris-style key-and-lock challenge.

A Hybrid 2D-3D Tangible Interface for Virtual Reality

A Hybrid 2D–3D Tangible Interface Combining a Smartphone and Controller for Virtual Reality

Simultaneous Real Walking and Asymmetric Input in Virtual Reality with a Smartphone-based Hybrid Interface.

Advancing Virtual Reality Interaction: A Ring-Shaped Controller and Pose Tracking

Tilt & Touch: Mobile Phone For 3d Interaction

Towards More Effective Interactions: A Hybrid Embodied Interaction Approach in Stereoscopic Displays

A touchable virtual screen interaction system with handheld Kinect camera

Is This My Phone? Operating a Physical Smartphone in Virtual Reality

SmartVR Pointer: Using Smartphones and Gaze Orientation for Selection and Navigation in Virtual Reality

Wearable teleoperation controller with 2-DoF robotic arm and haptic feedback for enhanced interaction in virtual reality

Above Surface Interaction for Multiscale Navigation in Mobile Virtual Reality

VRSmartphoneSketch: Augmenting VR Controller With A Smartphone For Mid-air Sketching

Virtual-Physical Interaction System

ARinVR: Bringing Mobile AR into VR

Haptic Handshank - A Handheld Multimodal Haptic Feedback Controller for Virtual Reality

A Lightweight and Efficient System for Tracking Handheld Objects in Virtual Reality

Haptic display for virtual reality: progress and challenges.

Holding Virtual Objects Using a Tablet for Tangible 3D Sketching in VR

Robot-enabled Tangible Virtual Assembly with Coordinated Midair Object Placement

3D Augmented Reality Tangible User Interface using Commodity Hardware

Intelligent Wearable Virtual Reality (VR) Gaming Controller for People with Motor Disabilities