Abstract:Design of process control scheme is critical for quality assurance to reduce variations in manufacturing systems. Taking semiconductor manufacturing as an example, extensive literature focuses on control optimization based on certain process models (usually linear models), which are obtained by experiments before a manufacturing process starts. However, in real applications, pre-defined models may not be accurate, especially for a complex manufacturing system. To tackle model inaccuracy, we propose a model-free reinforcement learning (MFRL) approach to conduct experiments and optimize control simultaneously according to real-time data. Specifically, we design a novel MFRL control scheme by updating the distribution of disturbances using Bayesian inference to reduce their large variations during manufacturing processes. As a result, the proposed MFRL controller is demonstrated to perform well in a nonlinear chemical mechanical planarization (CMP) process when the process model is unknown. Theoretical properties are also guaranteed when disturbances are additive. The numerical studies also demonstrate the effectiveness and efficiency of our methodology.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in complex manufacturing systems, traditional model - based process control methods are difficult to achieve precise quality control due to inaccurate models. Specifically, the paper focuses on how to design a model - free reinforcement learning (MFRL) process control scheme to reduce variations in the manufacturing process in semiconductor manufacturing, especially in the absence of an explicit process model. The paper proposes a model - free reinforcement learning control scheme combined with Bayesian inference (MFRL - BI) to update the distribution of perturbations in real - time, thereby reducing variations in the manufacturing process more effectively. The core problems of the paper are: 1. **Model inaccuracy**: In complex manufacturing systems, predefined models may not be accurate enough, especially for non - linear processes such as the chemical - mechanical planarization (CMP) process. This leads to difficulties in control optimization. 2. **Perturbation influence**: Perturbations are important factors affecting system output, but in existing MFRL methods, the influence of these perturbations is often ignored, resulting in poor control effects. 3. **Real - time data utilization**: How to use real - time data to optimize control strategies while reducing variations in system output. To address the above challenges, the paper proposes the MFRL - BI controller, which improves the basic MFRL controller in the following ways: - **Combined with Bayesian inference**: Use Bayesian inference to update the distribution of perturbations in real - time, thereby more accurately reflecting the actual manufacturing environment. - **Phased optimization**: Divide the control process into two phases: the optimization phase (Phase I) for learning control strategies, and the application phase (Phase II) for control optimization in online manufacturing. The paper also provides a theoretical analysis, proving that the proposed MFRL - BI controller is asymptotically optimal in theory, and verifies its effectiveness and efficiency through numerical studies.

MFRL-BI: Design of a Model-free Reinforcement Learning Process Control Scheme by Using Bayesian Inference

Batch-To-Batch Control of Batch Processes Based on Multilayer Recurrent Fuzzy Neural Network

Parameter Self-Tuning of SISO Compact-Form Model-Free Adaptive Controller Based on Long Short-Term Memory Neural Network

Accelerating Reinforcement Learning with Local Data Enhancement for Process Control

Reinforcement Learning for Process Control with Application in Semiconductor Manufacturing

Nonlinear adaptive control based on RBF networks and multi-model method

Online Reinforcement Learning Neural Network Controller Design for Nanomanipulation

Design Of Model Predictive Controller Based On Neural Systems

Learning-Based Neural Dynamic Surface Predictive Control for MMC

Robust LQ optimal controller designing for refining process

Accelerating reinforcement learning with case-based model-assisted experience augmentation for process control

Predictive Control of Voltage Source Inverter: an Online Reinforcement Learning Solution

A Deep Reinforcement Learning Approach to Improve the Learning Performance in Process Control

Opportunities of Hybrid Model-based Reinforcement Learning for Cell Therapy Manufacturing Process Control

Integrating process design and control using reinforcement learning

Human operator decision support for highly transient industrial processes: a reinforcement learning approach

Ai-based optimal control of fed-batch biopharmaceutical process leveraging deep reinforcement learning

A Model-Based Reinforcement Learning Approach for PID Design

Adaptive Pseudo Linear Rbf Model for Process Control

Optimal data-driven control of manufacturing processes using reinforcement learning: an application to wire arc additive manufacturing

Two-Dimensional Reinforcement Learning Model-Free Fault-Tolerant Control for Batch Processes Against Multi- Faults