Abstract:Smart manufacturing scheduling (SMS) requires a high degree of flexibility to successfully cope with changes in operational decision level planning processes in today's production environments, which are usually subject to high uncertainty. In such a unique and complex scenario as the real job shop, the modelling of SMS as a Markov decision process (MDP), and its approach by deep reinforcement learning (DRL), is a research field of growing interest given its characteristics. It allows us to consider achieving high flexibility levels by promoting process automation, autonomy in decision making, and the ability to act in real time when faced with disturbances and disruptions in a highly dynamic environment. This paper addresses the problem of scheduling a quasi-realistic job shop environment characterised by machines receiving jobs from buffers that accumulate numerous jobs using a wide variety of parts and multimachine routes with a diverse number of operation phases by developing a digital twin of the job shop based on a MDP with the DRL methodology. This is approached by: modelling the job shop scheduling environment with OpenAI Gym; designing an observation space with 18 job features; designing an action space composed of three priority heuristic rules; shaping a single reward function with a multi-objective characteristic; using the implementation of the proximal policy optimisation (PPO) algorithm from the Stable Baselines 3 library. This modelling approach, dubbed as job shop smart manufacturing scheduling (JS-SMS), is characterised by deterministic formulation and implementation. The model is subjected to validation by comparing it to several of the best-known heuristic priority rules. The main findings of this methodology allow to replicate, to a great extent, the positive aspects of heuristic rules and to mitigate the negative ones, which achieves more balanced behaviour in most of the measures established as performance indicators and outperforms heuristic rules from this multi-objective perspective. Finally, further research is oriented to dynamic and stochastic approaches to address the job shop reality in an Industry 4.0 context.

Real-time scheduling for a smart factory using a reinforcement learning approach

Real-time production scheduling using a deep reinforcement learning-based multi-agent approach

Dynamic scheduling for flexible job shop using a deep reinforcement learning approach

Hierarchical Reinforcement Learning for Multi-Objective Real-Time Flexible Scheduling in a Smart Shop Floor

Scalable Multi-agent Reinforcement Learning for Factory-wide Dynamic Scheduling

Reinforcement Learning with Composite Rewards for Production Scheduling in a Smart Factory.

Deep reinforcement learning for dynamic distributed job shop scheduling problem with transfers

Multi-agent Deep Reinforcement Learning for Dynamic Reconfigurable Shop Scheduling Considering Batch Processing and Worker Cooperation

Manufacturing Dispatching using Reinforcement and Transfer Learning

Dual Reinforcement Learning Architecture Based Scheduling in Reconfigurable Manufacturing Systems

Intelligent Scheduling of Discrete Automated Production Line Via Deep Reinforcement Learning

Job shop smart manufacturing scheduling by deep reinforcement learning

A Deep Multi-Agent Reinforcement Learning Approach to Solve Dynamic Job Shop Scheduling Problem

Digital twin and deep reinforcement learning enabled real-time scheduling for complex product flexible shop-floor

Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems

Multi-Agent Reinforcement Learning for Real-Time Dynamic Production Scheduling in a Robot Assembly Cell

Deep-Reinforcement-Learning-Based Production Scheduling in Industrial Internet of Things

Dynamic Job-Shop Scheduling Based on Transformer and Deep Reinforcement Learning

Deep Reinforcement Learning for Dynamic Flexible Job Shop Scheduling with Random Job Arrival

Digital Twin-Enabled Adaptive Scheduling Strategy Based on Deep Reinforcement Learning

Deep Reinforcement Learning-Based Dynamic Scheduling in Smart Manufacturing