Abstract:In-hand manipulation of tools using dexterous hands in real-world is an underexplored problem in the literature. In addition to more complex geometry and larger size of the tools compared to more commonly used objects like cubes or cylinders, task oriented in-hand tool manipulation involves many sub-tasks to be performed sequentially. This may involve reaching to the tool, picking it up, reorienting it in hand with or without regrasping to reach to a desired final grasp appropriate for the tool usage, and carrying the tool to the desired pose. Research on long-horizon manipulation using dexterous hands is rather limited and the existing work focus on learning the individual sub-tasks using a method like reinforcement learning (RL) and combine the policies for different subtasks to perform a long horizon task. However, in general a single method may not be the best for all the sub-tasks, and this can be more pronounced when dealing with multi-fingered hands manipulating objects with complex geometry like tools. In this paper, we investigate the use of a mixed-method approach to solve for the long-horizon task of tool usage and we use imitation learning, reinforcement learning and model based control. We also discuss a new RL-based teacher-student framework that combines real world data into offline training. We show that our proposed approach for each subtask outperforms the commonly adopted reinforcement learning approach across different subtasks and in performing the long horizon task in simulation. Finally we show the successful transferability to real world.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper attempts to address the problem of task-oriented long-duration operations using a dexterous hand for tool manipulation in the real world. Specifically, the paper focuses on the following aspects: 1. **Tool Manipulation**: Using a dexterous hand for tool manipulation in the real world is an under-researched problem. Compared to commonly used simple geometric objects (such as cubes or cylinders), tools have more complex geometries and larger sizes. 2. **Multi-task Sequential Execution**: Tool manipulation tasks often involve the sequential execution of multiple subtasks, such as: - Reaching the tool - Picking up the tool - Reorienting the tool in hand (which may require re-grasping) - Moving the tool to the desired location 3. **Limitations of Existing Methods**: Current research mainly focuses on using a single method (such as reinforcement learning) to learn each subtask and combining these strategies to complete long-duration tasks. However, a single method may not be suitable for all subtasks, especially when dealing with complex geometric tools using a multi-finger dexterous hand. ### Solution To overcome the above problems, the paper proposes a mixed-method approach that combines imitation learning, reinforcement learning, and model-based control. The specific contributions include: 1. **Mixed-Method Framework**: Each subtask is solved using the most suitable method, and these low-level strategies are then combined under a unified framework, with high-level strategy selection to execute the strategies. 2. **Teacher-Student Reinforcement Learning Framework**: A new teacher-student reinforcement learning method is proposed, incorporating sparse real-world data into offline training to improve the transfer performance from simulation to the real world. ### Experimental Validation The paper validates the effectiveness of the proposed method through experiments, including: 1. **Subtask Performance Comparison**: Ablation studies were conducted for each subtask (reaching, grasping, reorienting) to compare the performance of the proposed method with standard reinforcement learning methods. 2. **Overall Long-Duration Task Performance**: The overall performance of the proposed unified method was compared with low-level strategies trained using only reinforcement learning in a simulated environment. 3. **Real-World Deployment**: The proposed unified framework was deployed in the real world, verifying its effectiveness and high success rate in actual tasks. ### Conclusion The mixed-method framework proposed in the paper outperforms traditional reinforcement learning methods in both individual subtasks and overall long-duration tasks, and it successfully achieves tool manipulation tasks in the real world.

On the Feasibility of A Mixed-Method Approach for Solving Long Horizon Task-Oriented Dexterous Manipulation

Hybrid Learning- and Model-Based Planning and Control of In-Hand Manipulation

Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost

Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning

Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance

Enhancing Dexterity in Robotic Manipulation via Hierarchical Contact Exploration

Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition

Bi-DexHands: Towards Human-Level Bimanual Dexterous Manipulation

Learning Task-Oriented Dexterous Grasping from Human Knowledge

Contact-Implicit Model Predictive Control for Dexterous In-hand Manipulation: A Long-Horizon and Robust Approach

Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation

Comparison of Model-Based and Model-Free Reinforcement Learning for Real-World Dexterous Robotic Manipulation Tasks

A High-Efficient Reinforcement Learning Approach for Dexterous Manipulation

Hierarchical Tactile-Based Control Decomposition of Dexterous In-Hand Manipulation Tasks

Multi-Phase Multi-Objective Dexterous Manipulation with Adaptive Hierarchical Curriculum

Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation

Interactive learning for multi-finger dexterous hand: A model-free hierarchical deep reinforcement learning approach

Deep Dynamics Models for Learning Dexterous Manipulation