Abstract:Background Spiking neural network (SNN), as the next-generation neural network inspired by the human brain, has been proved to be promising for constructing energy-efficient systems due to its inherent property of dynamic representation and information processing mechanism. For applications in deep reinforcement learning (DRL), however, the actual performance of SNNs is usually weaker than that of deep neural networks in high-dimensional continuous control. Related studies have shown that SNN suffers from insufficient model representation capacity and ineffective parameter training methods. To advance SNN to the practical application level, we aim at challenging partially observable Markov decision process (POMDP) high-dimensional control problems. Method Based on Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3), we propose a Spiking Memory TD3 algorithm (SM-TD3), which is a hybrid training framework of a spiking Long Short-Term Memory (Spiking-LSTM) policy network and a deep critic network. The policy leverages population-encoding to improve input encoding precision, spiking-LSTM to provide memory function, and spatio-temporal backpropagation to train parameters. Results and Conclusions We use the Pybullet benchmark to test the performance of SM-TD3 and set up comparisons in three cases of full-observation Markov decision process (MDP), random noise, and random sensor missing. The results show that SM-TD3 with SNN energy-efficient framework solves the POMDP problems under large-scale and high-dimensional tasks. It reaches the same performance level as the deep LSTM-TD3 algorithm. At the same time, SM-TD3 still has competitive robustness when transferred to the same environment in different situations. Finally, we analyze the energy consumption of SM-TD3. Compared with the energy consumption of deep LSTM-TD3, the energy consumption of SM-TD3 is only 20% 50% of that. Facing the practical application level, the proposed SM-TD3 provides an effective solution for both high-performance and energy-efficient.

Brain-inspired Action Generation with Spiking Transformer Diffusion Policy Model

SDP: Spiking Diffusion Policy for Robotic Manipulation with Learnable Channel-Wise Membrane Thresholds

Spiking Memory Policy with Population-encoding for Partially Observable Markov Decision Process Problems

Vision-Action Semantic Associative Learning Based on Spiking Neural Networks for Cognitive Robot.

Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation

Diffusion policy: Visuomotor policy learning via action diffusion

SDiT: Spiking Diffusion Model with Transformer

Diffusion Transformer Policy

Prediction with Action: Visual Policy Learning via Joint Denoising Process

Motorsrnn: A Spiking Recurrent Neural Network Inspired by Brain Topology for the Effective and Efficient Decoding of Cortical Spike Trains

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation

Spiking-Diffusion: Vector Quantized Discrete Diffusion Model with Spiking Neural Networks

Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning

Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

Brain-inspired Computational Modeling of Action Recognition with Recurrent Spiking Neural Networks Equipped with Reinforcement Delay Learning

DNAct: Diffusion Guided Multi-Task 3D Policy Learning

Masked Spiking Transformer

The Ingredients for Robotic Diffusion Transformers