Abstract:Approaches to robotic manufacturing, assembly, and servicing of in-space assets range from autonomous operation to direct teleoperation, with many forms of semi-autonomous teleoperation in between. Because most approaches require one or more human operators at some level, it is important to explore the control and visualization interfaces available to those operators, taking into account the challenges due to significant telemetry time delay. We consider one motivating application of remote teleoperation, which is ground-based control of a robot on-orbit for satellite servicing. This paper presents a model-based architecture that: 1) improves visualization and situation awareness, 2) enables more effective human/robot interaction and control, and 3) detects task failures based on anomalous sensor feedback. We illustrate elements of the architecture by drawing on 10 years of our research in this area. The paper further reports the results of several multi-user experiments to evaluate the model-based architecture, on ground-based test platforms, for satellite servicing tasks subject to round-trip communication latencies of several seconds. The most significant performance gains were obtained by enhancing the operators' situation awareness via improved visualization and by enabling them to precisely specify intended motion. In contrast, changes to the control interface, including model-mediated control or an immersive 3D environment, often reduced the reported task load but did not significantly improve task performance. Considering the challenges of fully autonomous intervention, we expect that some form of teleoperation will continue to be necessary for robotic in-situ servicing, assembly, and manufacturing tasks for the foreseeable future. We propose that effective teleoperation can be enabled by modeling the remote environment, providing operators with a fused view of the real environment and virtual model, and incorporating interfaces and control strategies that enable interactive planning, precise operation, and prompt detection of errors.

Scientists and drivers in-the-loop, on-the-loop or out-of-the-loop? Holistic human systems integration of teleoperated vehicles

The work organization in teleoperated driving – a human reliability and socio-technical systems perspective

From Interaction to Cooperation: a new approach for human-machine interaction research for closing the out-of-the-loop unfamiliarity

Survey on Teleoperation Concepts for Automated Vehicles

Should Teleoperation Be like Driving in a Car? Comparison of Teleoperation HMIs

From Car-Driver-Handovers to Cooperative Interfaces: Visions for Driver–Vehicle Interaction in Automated Driving

Bridging system limits with human–machine-cooperation

Open Source Software for Teleoperated Driving

Systems-Theoretic Safety Assessment of Teleoperated Road Vehicles

Evaluation of Teleoperation Concepts to solve Automated Vehicle Disengagements

Steer with Me: A Predictive, Potential Field-Based Control Approach for Semi-Autonomous, Teleoperated Road Vehicles

Combined perception, control, and learning for teleoperation: key technologies, applications, and challenges

Teleoperation and Visualization Interfaces for Remote Intervention in Space

Human-centered design and evaluation of a workplace for the remote assistance of highly automated vehicles

On human-in-the-loop optimization of human–robot interaction

Fine Manipulation and Dynamic Interaction in Haptic Teleoperation

Trajectory Guidance: Enhanced Remote Driving of highly-automated Vehicles

An Agent-Based Discrete Event Simulation of Teleoperated Driving in Freight Transport Operations: The Fleet Sizing Problem

Modeling driver-vehicle interaction in automated driving

A novel testbed for investigating the impact of teleoperator dynamics on perceived environment dynamics