Abstract:Teleoperation tasks with multi-agent systems have a high potential in supporting human-swarm collaborative teams in exploration and rescue operations. However, it requires an intuitive and adaptive control approach to ensure swarm stability in a cluttered and dynamically shifting environment. We propose a novel human-swarm interaction system, allowing the user to control swarm position and formation by either direct hand motion or by trajectory drawing with a hand gesture interface based on the DNN gesture recognition. The key technology of the SwarmPaint is the user's ability to perform various tasks with the swarm without additional devices by switching between interaction modes. Two types of interaction were proposed and developed to adjust a swarm behavior: free-form trajectory generation control and shaped formation control. Two preliminary user studies were conducted to explore user's performance and subjective experience from human-swarm interaction through the developed control modes. The experimental results revealed a sufficient accuracy in the trajectory tracing task (mean error of 5.6 cm by gesture draw and 3.1 cm by mouse draw with the pattern of dimension 1 m by 1 m) over three evaluated trajectory patterns and up to 7.3 cm accuracy in targeting task with two target patterns of 1 m achieved by SwarmPaint interface. Moreover, the participants evaluated the trajectory drawing interface as more intuitive (12.9 %) and requiring less effort to utilize (22.7%) than direct shape and position control by gestures, although its physical workload and failure in performance were presumed as more significant (by 9.1% and 16.3%, respectively).

Natural Interactive Control Method for Unmanned Swarm Systems Based on Gesture Recognition

Web-based Interactive Drone Control Using Hand Gesture

Interaction and control with the auxiliary of hand gesture

End-Side Gesture Recognition Method for UAV Control

Online Hand Gesture Detection and Recognition for UAV Motion Planning

Research on Multi-modal Interactive Control for Quadrotor UAV

Gesture Recognition Based on Localist Attractor Networks with Application to Robot Control [Application Notes]

3D Intuitive Gesture Interaction via Motion Sensing

A Human-swarm Interaction Method Based on Augmented Reality

Personalized Human-Swarm Interaction through Hand Motion

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

SwarmPaint: Human-Swarm Interaction for Trajectory Generation and Formation Control by DNN-based Gesture Interface

An interaction system using mixed hand gestures.

HGIC: A Hand Gesture Based Interactive Control System for Efficient and Scalable Multi-UAV Operations

A gesture recognition system using Localist Attractor Networks for human-robot interaction

An Underwater Human–Robot Interaction Using Hand Gestures for Fuzzy Control

Virtual Interaction and Manipulation Control of a Hexacopter Through Hand Gesture Recognition from a Data Glove.

Hand Gesture Control for Human–Computer Interaction with Deep Learning

Gesture-based Piloting of an Aerial Robot using Monocular Vision

Comparative Analysis on Few Typical Algorithms of Gesture Recognition for Human-robot Interaction.

Towards Decentralized Human-Swarm Interaction by Means of Sequential Hand Gesture Recognition