Abstract:Independently exploring unknown spaces or finding objects in an indoor environment is a daily but challenging task for visually impaired people. However, common 2D assistive systems lack depth relationships between various objects, resulting in difficulty to obtain accurate spatial layout and relative positions of objects. To tackle these issues, we propose HIDA, a lightweight assistive system based on 3D point cloud instance segmentation with a solid-state LiDAR sensor, for holistic indoor detection and avoidance. Our entire system consists of three hardware components, two interactive functions~(obstacle avoidance and object finding) and a voice user interface. Based on voice guidance, the point cloud from the most recent state of the changing indoor environment is captured through an on-site scanning performed by the user. In addition, we design a point cloud segmentation model with dual lightweight decoders for semantic and offset predictions, which satisfies the efficiency of the whole system. After the 3D instance segmentation, we post-process the segmented point cloud by removing outliers and projecting all points onto a top-view 2D map representation. The system integrates the information above and interacts with users intuitively by acoustic feedback. The proposed 3D instance segmentation model has achieved state-of-the-art performance on ScanNet v2 dataset. Comprehensive field tests with various tasks in a user study verify the usability and effectiveness of our system for assisting visually impaired people in holistic indoor understanding, obstacle avoidance and object search.

A Novel 3D Sensory System for Robot-Assisted Mapping of Cluttered Urban Search and Rescue Environments

3-D Map Reconstruction in Cross-country Environment

Long-Range Traversability Awareness and Low-Lying Obstacle Negotiation with RealSense for the Visually Impaired

3D Map Building for Mobile Robot under Unknown Outdoor Environments

Coarse-to-fine Hybrid 3D Mapping System with Co-calibrated Omnidirectional Camera and Non-repetitive LiDAR

A Compact Handheld Sensor Package with Sensor Fusion for Comprehensive and Robust 3D Mapping

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Three-Dimensional Geometric-Physical Modeling of an Environment with an In-House-Developed Multi-Sensor Robotic System

Real-time 3D mapping using a 2D laser scanner and IMU-aided visual SLAM

P‐2.11: Research on Scene 3d Reconstruction Technology Based on Multi‐sensor Fusion

HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor

3-D relative positioning sensor for indoor flying robots

A Human–Robot Collaborative System for Robust Three-Dimensional Mapping

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

Multi-Sensor Integration for Indoor 3D Reconstruction

Autonomous Exploration of Unknown Indoor Environments for High-Quality Mapping Using Feature-Based RGB-D SLAM

Environment Mapping Using Sensor Fusion of 2D Laser Scanner and 3D Ultrasonic Sensor for a Real Mobile Robot

Efficient Exploration for Real-Time Robot Indoor 3d Mapping

HeteroFusion: Dense Scene Reconstruction Integrating Multi-Sensors.

Online Localisation and Colored Mesh Reconstruction Architecture for 3D Visual Feedback in Robotic Exploration Missions