Abstract:Assembly state recognition facilitates the execution of assembly procedures, offering feedback to enhance efficiency and minimize errors. However, recognizing assembly states poses challenges in scalability, since parts are frequently updated, and the robustness to execution errors remains underexplored. To address these challenges, this paper proposes an approach based on representation learning and the novel intermediate-state informed loss function modification (ISIL). ISIL leverages unlabeled transitions between states and demonstrates significant improvements in clustering and classification performance for all tested architectures and losses. Despite being trained exclusively on images without execution errors, thorough analysis on error states demonstrates that our approach accurately distinguishes between correct states and states with various types of execution errors. The integration of the proposed algorithm can offer meaningful assistance to workers and mitigate unexpected losses due to procedural mishaps in industrial settings. The code is available at: <a class="link-external link-https" href="https://timschoonbeek.github.io/state_rec" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on two aspects: 1. **Scalability issue**: In an industrial environment, one of the main challenges faced by assembly - state recognition is the frequent updating of parts. This means that every time any small change occurs in the parts or steps during the assembly process, it is necessary to relabel the data and retrain or fine - tune the entire neural network anew. This is not only costly but also greatly limits its scalability in practical industrial applications. 2. **Insufficient robustness to execution errors**: Existing methods perform poorly in dealing with undefined execution errors (i.e., errors not clearly specified during the training process). Since encountering unexpected states in complex assembly tasks is very common, existing methods cannot reliably detect these undefined error states. And these undefined and unpredictable errors are often the ones that cause the most trouble. To solve these problems, this paper proposes a method based on representation learning and introduces a new loss - function modification - Intermediate - State Informed Loss (ISIL). This method improves the generalization ability of the model to new assembly states and unseen execution errors by learning a function that can map images to a meaningful embedding space, rather than explicitly memorizing every possible assembly state. ### Main contributions - **Representation - learning method**: Proposed a representation - learning method for assembly - state recognition, which outperforms classification methods and also performs well on unseen part configurations. - **ISIL loss function**: Proposed a simple yet effective loss - function modification, ISIL, which uses unlabeled intermediate states as negative samples and significantly improves clustering and classification performance. - **Research on error states**: Conducted extensive research on incorrect assembly states, showing that the representation - learning - based method better identifies assembly errors in real - case scenarios than the classification - based method, even without training on data containing error states. Through these improvements, this method can handle small changes in the assembly process without retraining the model and can more accurately distinguish between correct states and various types of execution errors, thereby providing meaningful assistance to workers in an industrial environment and reducing unexpected losses caused by procedural mistakes.

Supervised Representation Learning towards Generalizable Assembly State Recognition

Learning-Based Hand Motion Capture and Understanding in Assembly Process

Learn to Differ: Sim2Real Small Defection Segmentation Network

Find the Assembly Mistakes: Error Segmentation for Industrial Applications

ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation

State Representation Learning for Effective Deep Reinforcement Learning.

Pluvio: Assembly Clone Search for Out-of-domain Architectures and Libraries through Transfer Learning and Conditional Variational Information Bottleneck

Unsupervised State Representation Learning in Atari

Probabilistic Graph Based Spatial Assembly Relation Inference for Programming of Assembly Task by Demonstration.

Hand-in-Hand Guidance: an Explore-Exploit Based Reinforcement Learning Method for Performance Driven Assembly-Adjustment

State Representation Learning with Adjacent State Consistency Loss for Deep Reinforcement Learning.

Domain adaptive segmentation method for mechanical assembly based on iterative loops

Towards Learning from Demonstration System for Parts Assembly: A Graph Based Representation for Knowledge

Hierarchical Hybrid Learning for Long-Horizon Contact-Rich Robotic Assembly

Real-time action localization of manual assembly operations using deep learning and augmented inference state machines

LLM-Empowered State Representation for Reinforcement Learning

Human-object integrated assembly intention recognition for context-aware human-robot collaborative assembly

State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

Deep multimodal representation learning for generalizable person re-identification

Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images

Reinforcement Learning with Decoupled State Representation for Robot Manipulations