Abstract:Background Spiking neural network (SNN), as the next-generation neural network inspired by the human brain, has been proved to be promising for constructing energy-efficient systems due to its inherent property of dynamic representation and information processing mechanism. For applications in deep reinforcement learning (DRL), however, the actual performance of SNNs is usually weaker than that of deep neural networks in high-dimensional continuous control. Related studies have shown that SNN suffers from insufficient model representation capacity and ineffective parameter training methods. To advance SNN to the practical application level, we aim at challenging partially observable Markov decision process (POMDP) high-dimensional control problems. Method Based on Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3), we propose a Spiking Memory TD3 algorithm (SM-TD3), which is a hybrid training framework of a spiking Long Short-Term Memory (Spiking-LSTM) policy network and a deep critic network. The policy leverages population-encoding to improve input encoding precision, spiking-LSTM to provide memory function, and spatio-temporal backpropagation to train parameters. Results and Conclusions We use the Pybullet benchmark to test the performance of SM-TD3 and set up comparisons in three cases of full-observation Markov decision process (MDP), random noise, and random sensor missing. The results show that SM-TD3 with SNN energy-efficient framework solves the POMDP problems under large-scale and high-dimensional tasks. It reaches the same performance level as the deep LSTM-TD3 algorithm. At the same time, SM-TD3 still has competitive robustness when transferred to the same environment in different situations. Finally, we analyze the energy consumption of SM-TD3. Compared with the energy consumption of deep LSTM-TD3, the energy consumption of SM-TD3 is only 20% 50% of that. Facing the practical application level, the proposed SM-TD3 provides an effective solution for both high-performance and energy-efficient.

A Scalable Model-Free Recurrent Neural Network Framework for Solving POMDPs

ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints

Recurrent Natural Policy Gradient for POMDPs

A fast algorithm for solving large scale nonlinear optimization problems using RNN

Recursively-Constrained Partially Observable Markov Decision Processes

Partially Observable Planning and Learning for Systems with Non-Uniform Dynamics

Deep Recurrent Policy Networks for Planning under Partial Observability.

Counterexample-Guided Strategy Improvement for POMDPs Using Recurrent Neural Networks

Real-Time Recurrent Reinforcement Learning

Multilevel Monte-Carlo for Solving POMDPs Online

GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL

Scaling Long-Horizon Online POMDP Planning via Rapid State Space Sampling

Spiking Memory Policy with Population-encoding for Partially Observable Markov Decision Process Problems

Making Better Decision by Directly Planning in Continuous Control

Recurrent Model Predictive Control: Learning an Explicit Recurrent Controller for Nonlinear Systems

Analytical Solution to A Discrete-Time Model for Dynamic Learning and Decision-Making

Scalable Model-based Policy Optimization for Decentralized Networked Systems

SVQN: Sequential Variational Soft Q-Learning Networks

On-Robot Bayesian Reinforcement Learning for POMDPs

On Improving Deep Reinforcement Learning for POMDPs