A Fuzzy Multi-Step Q Learning Algorithm Based On Q(Lambda)- Learning And Its Application

Yingfei Wang,Dingsheng Luo

2011-01-01

Abstract:In this paper, we propose a new learning algorithm that can deal with continuous sate spaces and continuous action spaces. It is based on a modified Q(lambda) learning algorithm and the inference result by a Q fuzzy rule-based system. The behavior of this algorithm has been demonstrated through computer simulations.

What problem does this paper attempt to address?

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

Long Yang,Minhao Shi,Qian Zheng,Wenjia Meng,Gang Pan

DOI: https://doi.org/10.24963/ijcai.2018/414

2018-01-01

Abstract:Recently, a new multi-step temporal learning algorithm Q(σ) unifies n-step Tree-Backup (when σ = 0) and n-step Sarsa (when σ = 1) by introducing a sampling parameter σ. However, similar to other multi-step temporal-difference learning algorithms, Q(σ) needs much memory consumption and computation time. Eligibility trace is an important mechanism to transform the off-line updates into efficient on-line ones which consume less memory and computation time. In this paper, we combine the original Q(σ) with eligibility traces and propose a new algorithm, called Qπ(σ,λ), where λ is trace-decay parameter. This new algorithm unifies Sarsa(λ) (when σ = 1) and Qπ (λ) (when σ = 0). Furthermore, we give an upper error bound of Qπ(σ,λ) policy evaluation algorithm. We prove that Qπ (σ, λ) control algorithm converges to the optimal value function exponentially. We also empirically compare it with conventional temporal-difference learning methods. Results show that, with an intermediate value of σ, Qπ(σ,λ) creates a mixture of the existing algorithms which learn the optimal value significantly faster than the extreme end (σ = 0, or 1).
Adaptive Double Fuzzy Systems Based Q-Learning for Pursuit-Evasion Game

Shuaizheng Liu,Xiaoxiang Hu,Kejun Dong

DOI: https://doi.org/10.1016/j.ifacol.2022.05.044

2022-01-01

IFAC-PapersOnLine

Abstract:The paper addresses the problem of obtaining the optimal solutions of pursuit and evasion game in continuous time system. A novel technique that combines double fuzzy systems with Q-learning is proposed. The system consists of two segments: fuzzy Q-learning (FQL) module and Q-value table fuzzy inference system (QTFIS). The linguistic rules are updated adaptively by Q-learning in FQL module. Local Q-value table which takes advantages of the sparsity of Q-learning is the kernel data structure in FQL. The Controllers of pursuer and evader are provided by LQT while the parameters of fuzzy inference system are updated adaptively. Q-value function is approximated by the fuzzy inference system according to QTFIS. The innovative method has an advantage of not relying on accurate controlled models. The difficult drawback of continuous state space and continuous actions for Q-learning is overcome by the novel technique. The limitation of low dimensional space for Q-learning is broken by the system. The proposed technique is applied to an imperfect information pursuit-evasion game between UAVs. Finally, computer simulations show the usefulness of the creative approach.
A Novel Q-Learning Approach with Continuous States and Actions

Yi Zhou,Meng Joo Er

DOI: https://doi.org/10.1109/cca.2007.4389199

2007-01-01

Abstract:This paper presents a generalized Q-learning method termed dynamic fuzzy continuous-action Q-learning (DFCAQ) that works in continuous domains. It can be regarded as an extension of Millan's work in continuous-action Q-learning. In the DFCAQ approach, continuous states and actions are generated via a fuzzy structure. Instead of considering actions selected by the nearest unit only in the original continuous-action Q-learning, the global action is generated via a fuzzy approach. Compared with Jouffe's fuzzy Q-learning, the DFCAQ fuzzy structure can be automatically and dynamically generated. At the same time, the local actions in the DFCAQ method are average values of the discrete actions weighted by their Q-values. In addition, comparison studies in robotics domains show the superiority of the proposed DFCAQ method.
A Multi-step Q Reinforcement Learning Algorithm

CHEN Sheng-Lei,WU Hui-Zhong,HAN Xiang-Lan,XIAO Liang

2006-01-01

Computer Science

Abstract:Q learning is of great importance in reinforcement learning.MQ algorithm with multi-step predicting capabil- ity is proposed to compensate the drawbacks of Q learning and Q(γ)algorithmin in this paper .Firsly MDP model is presented.Then based on the analyses of Q learning and Q(γ)algorithm,MQ algorithm is proposed.The algorithm's update strategy and determination rule of k are also analyzed.The effectiveness of this algorithm is verified through the cliff walking simulation experiments.Theoretical analyses and experiments indicate that better predicting capability and decreased computation complexity can be obtained in this algorithm.So it can balance update speed and complexity ef- fectively.
Multi-Step Truncated Q Learning Algorithm

Shenglei Chen,HuiZhong Wu,Xianglan Han,Liang Xiao

DOI: https://doi.org/10.1109/icmlc.2005.1526943

2005-01-01

Abstract:Q learning is of great importance in reinforcement learning. To compensate the drawbacks of Q learning and Q(lambda) algorithm, MTQ algorithm is proposed in this paper. It makes use of future information of k steps to update current Q value. Thus it can consider more long-term benefit and the computation complexity is also decreased. Good balance is achieved between update speed and computation complexity. Experiments demonstrate effectiveness of this algorithm.
Multi-Agent Reward-Iteration Fuzzy Q-Learning

Leng Lixiong,Li Jingchen,Zhu Jinhui,Hwang Kao-Shing,Shi Haobin

DOI: https://doi.org/10.1007/s40815-021-01063-4

IF: 4.085

2021-01-01

International Journal of Fuzzy Systems

Abstract:Fuzzy Q-learning extends Q-learning to continuous state space and has been applied to a wide range of applications such as robot control. But in a multi-agent system, the non-stationary environment makes joint policy challenging to converge. To give agents more suitable rewards in a multi-agent environment, a multi-agent reward-iteration fuzzy Q-learning (RIFQ) is proposed for multi-agent cooperative tasks. The state space is divided into three channels by the proposed state-divider with fuzzy logic. The reward of an agent is reshaped iteratively according to its state, and the update sequence is constructed by calculating the relation among states of different agents. Then, the value functions are updated top-down. By replacing the reward given by the environment with the reshaped reward, agents can avoid the most unreasonable punishments and receive rewards selectively. RIFQ provides a feasible reward relationship for multi-agents, which makes the training of multi-agent more steady. Several simulation experiments show that RIFQ is not limited by the number of agents and has a faster convergence speed than baselines.
Dynamic Fuzzy Q-Learning and Its Real-Time Application in Embedded System

LU Yong-Kui,XU Min,LI Yong-Xin,DU Hua-Sheng,WU Yue-Hua,Jie Yang

DOI: https://doi.org/10.3969/j.issn.1003-6059.2006.04.002

2006-01-01

Pattern Recognition and Artificial Intelligence

Abstract:A new dynamic fuzzy Q-learning (DFQL) method is presented in this paper which is capable of tuning fuzzy inference systems (FIS) online. In DFQL system, the generation of continuous actions depends upon a discrete number of actions of every fuzzy rule and the vector of firing strengths of fuzzy rule. In order to explore the set of possible actions and acquire experiences through the reinforcement signals, the actions are selected using an exploration-exploitation strategy based on the expended greedy algorithm. A function Q that gives the action quality with eligibility trace and meta learning rule is used here to speed up learning, e-completeness of fuzzy rules criterion and temporal-difference (TD) error criterion are considered for rule generation. The DFQL approach has been applied to a real-time control caterpillar robot for the wall following task. Experimental results and comparative studies with the fuzzy Q-learning and continuous-action Q-learning in the wall-following task of mobile robots demonstrate that the proposed DFQL method is superior.
Automatic Generation Of Fuzzy Inference Systems Using Incremental-Topological-Preserving-Map-Based Fuzzy Q-Learning

Meng Joo Er,Linn San

DOI: https://doi.org/10.1109/FUZZY.2008.4630410

2008-01-01

Abstract:This paper represents a new approach for automatically generating Fuzzy Inference System (FIS) using Incremental Topological Preserving Map Fuzzy Q-Learning (ITPM-FQL). The ITPM-FQL can create and tune the fuzzy rules automatically without any priori knowledge. The online self organizing ITPM approach is used to achieve automatic structure identification while the Fuzzy Q-Learning approach is used for parameter identification to deal with continuous states and actions. Compared with the first author's previous works in Dynamic Fuzzy Q-Learning (DFQL), this proposed approach is able to achieve fewer numbers of fuzzy rules. Similar to the DFQL epsilon-completeness criterion is used to generate fuzzy rules but the convergence capability of the ITPM is added to provide flexible fuzzy clustering. Experimental results and comparative studies with conventional Fuzzy Q-Learning (FQL), Continuous-Action Q-Learning (CAQL), DFQL and its related developments, Dynamic Self Generated Fuzzy Q-Learning (DSGFQL) and Enhanced Dynamic Self Generated Fuzzy Q.-Learning (EDSGFQL), in wall-following task of a mobile robot are presented to demonstrate the superiority of the proposed approach.
A Q-learning Method for Continuous Space Based on Self-organizing Fuzzy RBF Network

CHENG Yu-hu,WANG Xue-song,YI Jian-qiang,SUN Wei

DOI: https://doi.org/10.3969/j.issn.1002-0411.2008.01.001

2008-01-01

Information and Control

Abstract:For reinforcement learning control in continuous spaces,a Q-learning method based on a self-organizing fuzzy RBF(radial basis function) network is proposed.Input of the fuzzy RBF network is state,and the outputs are continuous actions and the corresponding Q-values,which realizes the mapping from a continuous state space to a continuous action space.At first,the continuous action space is discretized into the discrete actions with definite number,and a completely greedy policy is used to select a discrete action with the maximum Q-value as the winning local actions of each fuzzy rule.Then a command fusion mechanism is adopted to weight the winning local actions of each fuzzy rule according to its utility value,and a continuous action is generated for the actual system.Moreover,in order to simplify the network structure and improve the learning speed,an improved resource allocating network(RAN) algorithm and a gradient descent algorithm are applied to adjust the structure and parameters of the fuzzy RBF network in an on-line and adaptive manner respectively.The effectiveness of the proposed Q-learning method is shown through simulation on the balancing control of an inverted pendulum system.
Automatic generation of fuzzy inference systems by dynamic fuzzy Q-learning

Chang Deng,Meng Joo Er

DOI: https://doi.org/10.1109/ICSMC.2003.1244384

2003-01-01

Abstract:This paper presents a dynamic Q-learning (DFQL) method that is capable of tuning the fuzzy inference systems (FIS) online. On-line self-organizing learning is developed so that structure and parameters identification are accomplished automatically and simultaneously based only on Q-learning. Self-organizing fuzzy inference is introduced to calculate actions and Q-functions so as to enable us to deal with continuous-valued states and actions. Fuzzy rules provide a natural mean to incorporate the bias components for rapid reinforcement learning. Experimental results and comparative studies with the fuzzy Q-learning the wall following task of mobile robots demonstrate the superiority of the proposed DFQL method.
A PROPOSAL OF WEIGHTED Q-LEARNING FOR CONTINUOUS STATE AND ACTION SPACES

Yuhu Cheng,Jianqiang Yi,Dongbin Zhao

2005-01-01

Abstract:A kind of weighted Q-Learning algorithm suitable for control systems with continuous state and action spaces was proposed. The hidden layer of RBF network was designed dynamically by virtue of the proposed modified growing neural gas algorithm so as to realize the adaptive understanding of the continuous state space. Based on the standard Q-Learning implemented by RBF network, the weighted Q-Learning was used to solve the control problem with continuous action outputs. Simulation result of mountain car control verified the validity of the proposed weighted Q-Learning algorithm. Copyright © 2005 IFAC
Real-time Dynamic Fuzzy Q-learning and Control of Mobile Robots

Chang Deng,Meng Joo Er

2003-01-01

Abstract:In this paper, a dynamic fuzzy Q-learning (DFQL) method capable of navigating a mobile robot efficiently is presented. The fuzzy rules for navigation can be generated and tuned automatically based on Q-learning. Continuous-valued states and actions are handled using fuzzy reasoning. Prior knowledge can be embedded into the fuzzy rules for rapid and safe learning. The eligibility trace method is employed in our algorithm, leading to faster learning and alleviating the experimentation-sensitive problem where an arbitrarily bad training policy might result in a non-optimal policy. Experimental results demonstrate that the robot is able to learn the appropriate navigation policy with a few trials.
A kind of weighted Q-learning for continuous state and action spaces

Yu-hu CHENG,Jian-qiang YI,Xue-song WANG,Dong-bin ZHAO

DOI: https://doi.org/10.3969/j.issn.1007-449X.2005.06.013

2005-01-01

Abstract:Aiming at control problems of continuous state and action spaces,a kind of weighted Q-learning was proposed to solve the application under the condition of continuous state and action spaces.The hidden layer of RBF network was designed dynamically by virtue of the modified growing neural gas algorithm so as to realize the adaptive understanding of the continuous state space.Based on the standard Q-learning implemented by RBF network,the weighted Q-learning was used to solve the control problem with continuous action outputs.Simulation result verified the validity of the algorithm.
Q learning based on self-organizing fuzzy radial basis function network

Xuesong Wang,Yuhu Cheng,Wei Sun

DOI: https://doi.org/10.1007/11759966_90

2006-01-01

Abstract:A fuzzy Q learning based on a self-organizing fuzzy radial basis function (FRBF) network is proposed to solve the ‘curse of dimensionality’ problem caused by state space generalization in the paper. A FRBF network is used to represent continuous action and the corresponding Q value. The interpolation technique is adopted to represent the appropriate utility value for the wining local action of every fuzzy rule. Neurons can be organized by the FRBF network itself. The methods of the structure and parameter learning, based on new adding and merging neurons techniques and a gradient descent algorithm, are simple and effective, with a high accuracy and a compact structure. Simulation results on balancing control of inverted pendulum illustrate the performance and applicability of the proposed fuzzy Q learning scheme to real-world problems with continuous states and continuous actions.
Dynamic fuzzy Q-learning and control of mobile robots

Deng, C.,Er, M.J.,Xu, J.

DOI: https://doi.org/10.1109/ICARCV.2004.1469797

2004-01-01

Abstract:In this paper, a dynamic fuzzy Q-learning (DFQL) method navigating a mobile robot efficiently is presented. Self-organizing fuzzy inference is introduced to calculate actions and Q-functions which capable of enabling us to deal with continuous-valued states and actions. Consequently, fuzzy rules can be generated automatically. Fuzzy inference systems provide a natural mean of incorporating the bias components for rapid reinforcement learning. Furthermore, the eligibility trace method is employed in our algorithm, leading to faster learning and alleviating the experimentation-sensitive problem where an arbitrarily bad training policy might result in a non-optimal policy. Experimental results demonstrate that the robot is able to learn the right policy with a few trials.
Double Q($σ$) and Q($σ, λ$): Unifying Reinforcement Learning Control Algorithms

Markus Dumke

DOI: https://doi.org/10.48550/arXiv.1711.01569

2017-11-05

Abstract:Temporal-difference (TD) learning is an important field in reinforcement learning. Sarsa and Q-Learning are among the most used TD algorithms. The Q($\sigma$) algorithm (Sutton and Barto (2017)) unifies both. This paper extends the Q($\sigma$) algorithm to an online multi-step algorithm Q($\sigma, \lambda$) using eligibility traces and introduces Double Q($\sigma$) as the extension of Q($\sigma$) to double learning. Experiments suggest that the new Q($\sigma, \lambda$) algorithm can outperform the classical TD control methods Sarsa($\lambda$), Q($\lambda$) and Q($\sigma$).

Artificial Intelligence,Machine Learning
Online Tuning of Fuzzy Inference Systems Using Dynamic Fuzzy Q-Learning

MJ Er,C Deng

DOI: https://doi.org/10.1109/tsmcb.2004.825938

2004-01-01

Abstract:This paper presents a dynamic fuzzy Q-learning (DFQL) method that is capable of tuning fuzzy inference systems (FIS) online. A novel online self-organizing learning algorithm is developed so that structure and parameters identification are accomplished automatically and simultaneously based only on Q-learning. Self-organizing fuzzy inference is introduced to calculate actions and Q-functions so as to enable us to deal with continuous-valued states and actions. Fuzzy rules provide a natural mean of incorporating the bias components for rapid reinforcement learning. Experimental results and comparative studies with the fuzzy Q-learning (FQL) and continuous-action Q-learning in the wall-following task of mobile robots demonstrate that the proposed DFQL method is superior.
Discrete-time optimal control scheme based on Q-learning algorithm

Qinglai Wei,Derong Liu,Ruizhuo Song

DOI: https://doi.org/10.1109/ICICIP.2016.7885888

2016-01-01

Abstract:This paper is concerned with optimal control problems of discrete-time nonlinear systems via a novel Q-learning algorithm. In the newly developed Q-learning algorithm, the iterative Q function in each iteration is required to update on the whole state and control spaces, instead of being updated by a single state and control pair. A new convergence criterion of the corresponding Q-learning algorithm is presented, where the traditional constraints for the learning rates of Q-learning algorithms is relaxed. Finally, simulation results are provided to exemplify the good performance of the developed algorithm.
Q-Learning for Continuous State Space Based on a Support Vector Machine

WANG Xue-song,TIAN Xi-lan,CHENG Yu-hu

DOI: https://doi.org/10.3321/j.issn:1000-1964.2008.01.019

2008-01-01

Abstract:Aiming at the reinforcement learning controller design under continuous state space, a new Q-learning method based on a support vector machine (SVM) was proposed. A SVM is used to estimate the Q-value of state-action pair on-line because a SVM can not fall into local minimum and is of perfect generalization property, by which the 'curse of dimension' easily caused by state space generalization was solved. A sliding time-window was introduced to realize the on-line learning of SVM. If the detected data contains no new information, the sampling set of SVM keeps unchanged. On the other hand, if the data contains new information, the time-window should be slid to update the sampling set and so as to update the regression model of SVM. Different weights were assigned to the data in the time-window in order to fully exploit the data information. Computer simulation results on cart-pole balancing control show that the proposed Q-learning method is suitable for reinforcement learning control for nonlinear system with continuous state.
Value approximation with least squares support vector machine in reinforcement learning system

WANG Xue-song,TIAN Xi-lan,CHENG Yu-hu,MA Xiao-ping

2007-01-01

Journal of Computational and Theoretical Nanoscience

Abstract:In this paper, we propose a new Q learning system based on the least squares support vector machine (LS-SVM) for continuous state space. A least squares support vector machine is used to realize a mapping from state-action pair to Q value function. A sliding time window mechanism and a criterion of adding the new observed data to the training sample set of LS-SVM are introduced into the method. Based on the on-line estimated Q values of all the state-action pairs from LS-SVM, a stochastic action selector is used to generate stochastically an actual action according to Boltzmann-Gibbs distribution of Q values. In order to improve the learning speed, a simulated annealing method is used to adjust the temperature value of Boltzmann-Gibbs distribution dynamically. The simulation results of Mountain Car Control show that the proposed Q learning method has characteristics of a simple control structure and high learning efficiency. Copyright © 2007 American Scientific Publishers. All rights reserved.

A Fuzzy Multi-Step Q Learning Algorithm Based On Q(Lambda)- Learning And Its Application

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

Adaptive Double Fuzzy Systems Based Q-Learning for Pursuit-Evasion Game

A Novel Q-Learning Approach with Continuous States and Actions

A Multi-step Q Reinforcement Learning Algorithm

Multi-Step Truncated Q Learning Algorithm

Multi-Agent Reward-Iteration Fuzzy Q-Learning

Dynamic Fuzzy Q-Learning and Its Real-Time Application in Embedded System

Automatic Generation Of Fuzzy Inference Systems Using Incremental-Topological-Preserving-Map-Based Fuzzy Q-Learning

A Q-learning Method for Continuous Space Based on Self-organizing Fuzzy RBF Network

Automatic generation of fuzzy inference systems by dynamic fuzzy Q-learning

A PROPOSAL OF WEIGHTED Q-LEARNING FOR CONTINUOUS STATE AND ACTION SPACES

Real-time Dynamic Fuzzy Q-learning and Control of Mobile Robots

A kind of weighted Q-learning for continuous state and action spaces

Q learning based on self-organizing fuzzy radial basis function network

Dynamic fuzzy Q-learning and control of mobile robots

Double Q($σ$) and Q($σ, λ$): Unifying Reinforcement Learning Control Algorithms

Online Tuning of Fuzzy Inference Systems Using Dynamic Fuzzy Q-Learning

Discrete-time optimal control scheme based on Q-learning algorithm

Q-Learning for Continuous State Space Based on a Support Vector Machine

Value approximation with least squares support vector machine in reinforcement learning system