Abstract:Q-learning is widely recognized as an effective approach for synthesizing controllers to achieve specific goals. However, handling challenges posed by continuous state-action spaces remains an ongoing research focus. This paper presents a systematic analysis that highlights a major drawback in space discretization methods. To address this challenge, the paper proposes a symbolic model that represents behavioral relations, such as alternating simulation from abstraction to the controlled system. This relation allows for seamless application of the synthesized controller based on abstraction to the original system. Introducing a novel Q-learning technique for symbolic models, the algorithm yields two Q-tables encoding optimal policies. Theoretical analysis demonstrates that these Q-tables serve as both upper and lower bounds on the Q-values of the original system with continuous spaces. Additionally, the paper explores the correlation between the parameters of the space abstraction and the loss in Q-values. The resulting algorithm facilitates achieving optimality within an arbitrary accuracy, providing control over the trade-off between accuracy and computational complexity. The obtained results provide valuable insights for selecting appropriate learning parameters and refining the controller. The engineering relevance of the proposed Q-learning based symbolic model is illustrated through two case studies.

Logarithmic Quantization based Symbolic Abstractions for Nonlinear Control Systems

On stochastic adaptive control under quantization

OutputFeedbackControl forUncertainNonlinear Systems with InputQuantization ⋆

Symbolic Abstractions for Nonlinear Control Systems via Feedback Refinement Relation

Data-Driven Control of Linear Systems via Quantized Feedback

Data-driven Control of Unknown Linear Systems via Quantized Feedback

A New Adaptive Control Scheme for Uncertain Nonlinear Systems with Quantized Input Signal

Quantized stabilizing control of state‐dependent switching affine systems with control‐input and state‐measurement quantization

Adaptive Output Feedback Regulation for a Class of Nonlinear Systems Subject to Input and Output Quantization

Iterative learning fault-tolerant control of networked systems with quantitative sampling

Logic-Based Switching Control for Stabilization of Stochastic Feedforward Nonlinear Systems

Practical Adaptive Fuzzy Control of Nonlinear Pure-Feedback Systems with Quantized Nonlinearity Input

ℒ1adaptive Controller for a Class of Non-Affine Multi-Input Multi-Output Nonlinear Systems

Network-Based Quantized Control for Fuzzy Singularly Perturbed Semi-Markov Jump Systems and Its Application.

Attainability of the Minimum Data Rate for Stabilization of Linear Systems Via Logarithmic Quantization.

Inverse Optimal Adaptive Fuzzy Output Feedback Control for Nonlinear Systems with Output Quantization

A Simplified Fuzzy Wavelet Neural Control for Nonlinear Systems With Quantized Inputs and Deferred Constraints

Finite-Level Quantized Feedback Control for Linear Systems

Adaptive quantized control for uncertain nonlinear systems with unknown control directions

How to discretize continuous state-action spaces in Q-learning: A symbolic control approach

Zonotope-based Symbolic Controller Synthesis for Linear Temporal Logic Specifications