Abstract:The real-time monitoring of assembly operations in manufacturing industries can be used for manufacturing process optimization, which is crucial to manufacturers. It helps to improve productivity by automatically identifying the bottleneck and enhancing product quality by detecting errors and providing feedback to rectify them in real-time. However, developing a robust and reliable assembly monitoring system is not trivial due to the varying length of fine-grained assembly steps and anthropometric variations associated with assembly workers. To tackle the challenge, conventionally, wireless body-worn sensors have been used, leading to operator discomfort, and raising safety concerns. In this work, we propose a novel technique to automatically recognize and localize the assembly steps in real time called the State Machine Integrated Recognition and Localization (SMIRL). SMIRL is comprised of an inference machine and a state machine. They are responsible for detecting and localizing the actions, respectively. SMIRL can measure the duration of individual assembly steps and the entire cycle. Additionally, SMIRL can also detect the mistakes that may occur in an assembly and raise an alert to notify the operators in real time. The mistakes here can be breaking the predefined assembly sequence (Sequence Break) or missing any of the assembly steps (Missed steps). The effectiveness of SMIRL was evaluated against two datasets, with one being from an assembly workstation in industry and the other from a laboratory. The result shows that SMIRL can detect and localize the actions with an Intersection over Union (IoU) score of 87.53%, and identify Sequence Breaks and Missed Steps with F1-Scores of 86.64% and 87.45%, respectively. Through this study, we aim to contribute towards real-time monitoring of human-centric assembly operations to facilitate smart manufacturing.

Real-time Monitoring for Manual Operations with Machine Vision in Smart Manufacturing

Learning-Based Hand Motion Capture and Understanding in Assembly Process

A Versatile Machine Vision Algorithm for Real-Time Counting Manually Assembled Pieces

Advanced Monitoring of Manufacturing Process through Video Analytics

Online Measurement of Machining Tool Wear Based on Machine Vision

Computer Vision-Based Personnel Management in Complex Scenarios of Industrial Enterprises

A Real-Time Monitoring and Diagnosis System for Manufacturing Automation

Discrete-Event Shop-Floor Monitoring System in Rfid-Enabled Manufacturing

A Machine Vision Method for Non-Contact Tool Wear Inspection

Study and Application of Automatic Count System Based on Computer Vision Detect Technology

Research and Application of Visualized Real-Time Monitoring System for Complex Product Manufacturing Process

REAL-TIME RECOGNITION SYSTEM FOR MULTI-OBJECT PARTS IN RAPID MOVING BACKGROUND

Real-time action localization of manual assembly operations using deep learning and augmented inference state machines

Self-learning Time-varying Digital Twin System for Intelligent Monitoring of Automatic Production Line

YOLO V3+VGG16-based Automatic Operations Monitoring and Analysis in a Manufacturing Workshop under Industry 4.0

A Study on a General Cyber Machine Tools Monitoring System in Smart Factories

Research on Monitoring and Operation Evaluation Technology of Production Line Driven by Real-Time Data From Shop Floor

Human motion segmentation and recognition using machine vision for mechanical assembly operation

Research and Practice of Intelligent Scheduling Algorithm Based on Machine Vision in Mechanical Equipment Manufacturing

A Low-Cost Vision-Based Monitoring of Computer Numerical Control (CNC) Machine Tools for Small and Medium-Sized Enterprises (SMEs)

Tool wear monitoring in roughing and finishing processes based on machine internal data