Corrigendum: SCTN: event-based object tracking with energy-efficient deep convolutional spiking neural networks

Mingcheng Ji,Ziling Wang,Rui Yan,Qingjie Liu,Shu Xu,Huajin Tang

DOI: https://doi.org/10.3389/fnins.2023.1204334

IF: 4.3

2023-05-16

Frontiers in Neuroscience

Abstract:[This corrects the article DOI: 10.3389/fnins.2023.1123698.].

neurosciences

What problem does this paper attempt to address?

Corrigendum: A brain-inspired decision-making spiking neural network and its application in unmanned aerial vehicle

Feifei Zhao,Yi Zeng,Bo Xu

DOI: https://doi.org/10.3389/fnbot.2022.1092428

IF: 3.493

2022-11-29

Frontiers in Neurorobotics

Abstract:[This corrects the article DOI: 10.3389/fnbot.2018.00056.].

robotics,computer science, artificial intelligence,neurosciences
RETRACTED: Multiscale fast correlation filtering tracking algorithm based on a feature fusion model (Retracted article. See vol. 34, 2022)

Yuantao Chen,Jin Wang,Songjie Liu,Xi Chen,Jie Xiong,Jingbo Xie,Kai Yang

DOI: https://doi.org/10.1002/cpe.5533

2021-01-01

Abstract:In scenes high in visual complexity, the identification of a moving object can be affected by changes in scale and occlusion factors during the tracking process, resulting in reduced tracking accuracy. Accordingly, to address the problem of low accuracy, a multiscale fast correlation filtering tracking algorithm based on a feature fusion model is proposed in the present work with the aim of reducing the poor tracking effects caused by occlusion discrimination and scale changes in complex scenes. The object's grayscale (GRAY) feature, histogram of oriented gradient (HOG) feature, and color name (CN) feature are reduced to dimensions and fused to form a feature matrix. Moreover, a hierarchical principal component analysis (HPCA) algorithm is used to extract visually salient features and reconstruct the feature matrix under real-time conditions, the correlation filtering position is trained, the number of dimensions is effectively reduced, and the feature fusion matrix is used to train the multiscale fast correlation filtering, with the result that the object's position and scale can be accurately predicted. The proposed algorithm is then compared with five popular correlation filtering tracking algorithms. Experimental results demonstrate that its average tracking speed reaches a reasonable frames/second velocity; moreover, it can also achieve promising object tracking results on the OTB benchmark data sets. The tracking accuracy is superior to that of the other five correlation filtering tracking algorithms when applied to scenes featuring object occlusion and changes in scale. The proposed algorithm also exhibits better robustness and improved performance under real-time conditions in sophisticated scenarios, including scale variation, deformation, fast motion, occlusion, and so on.
Spiking SiamFC++: Deep Spiking Neural Network for Object Tracking

Shuiying Xiang,Tao Zhang,Shuqing Jiang,Yanan Han,Yahui Zhang,Chenyang Du,Xingxing Guo,Licun Yu,Yuechun Shi,Yue Hao

DOI: https://doi.org/10.48550/arXiv.2209.12010

2022-09-24

Abstract:Spiking neural network (SNN) is a biologically-plausible model and exhibits advantages of high computational capability and low power consumption. While the training of deep SNN is still an open problem, which limits the real-world applications of deep SNN. Here we propose a deep SNN architecture named Spiking SiamFC++ for object tracking with end-to-end direct training. Specifically, the AlexNet network is extended in the time domain to extract the feature, and the surrogate gradient function is adopted to realize direct supervised training of the deep SNN. To examine the performance of the Spiking SiamFC++, several tracking benchmarks including OTB2013, OTB2015, VOT2015, VOT2016, and UAV123 are considered. It is found that, the precision loss is small compared with the original SiamFC++. Compared with the existing SNN-based target tracker, e.g., the SiamSNN, the precision (succession) of the proposed Spiking SiamFC++ reaches 85.24% (64.37%), which is much higher than that of 52.78% (44.32%) achieved by the SiamSNN. To our best knowledge, the performance of the Spiking SiamFC++ outperforms the existing state-of-the-art approaches in SNN-based object tracking, which provides a novel path for SNN application in the field of target tracking. This work may further promote the development of SNN algorithms and neuromorphic chips.

Computer Vision and Pattern Recognition,Artificial Intelligence
Spike Trains Encoding and Threshold Rescaling Method for Deep Spiking Neural Networks

Yang Xu,Huajin Tang,Jinwei Xing,Hongying Li

DOI: https://doi.org/10.1109/ssci.2017.8285427

2017-01-01

Abstract:Deep neural networks such as convolutional neural networks (CNNs) have achieved a great success in a broad range of fields. Spiking neural networks (SNNs) are designed as a solution for realizing ultra-low power consumption using spikebased neuromorphic hardware. To enable the mapping between conventional artificial neural networks (ANNs) and spike-based neuromorphic hardware, direct conversion from conventional ANNs into SNNs has recently been proposed. However, performance loss is hard to avoid after conversion. To reduce the loss, we analyze the encoding methods of SNNs and the optimization methods for converting. We use the rate coding and approximate the output of an activation function of CNNs by the number of spikes produced by a neuron within a given time window in SNNs. We propose a method for generating fixed uniform spike train whose number of spikes is as exact as we expect, and present an optimization method to reduce the loss by rescaling the threshold of each layer in SNNs. For evaluation, we train three different CNNs with MNIST and SVHN datasets separately and convert all of them into SNNs. The networks achieve a max accuracy of 99.17% on MNIST and of 93.36% on additional training dataset of SVHN. The results show the proposed fixed and uniform spike train not only outperforms the Poisson distributed spike train but also costs less time to generate. Our threshold rescaling method greatly improves the performance of SNNs.
Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Qi Xu,Yaxin Li,Jiangrong Shen,Pingping Zhang,Jian K. Liu,Huajin Tang,Gang Pan

DOI: https://doi.org/10.1109/tnnls.2022.3232106

IF: 14.255

2024-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Thanks to their event-driven nature, spiking neural networks (SNNs) are surmised to be great computation-efficient models. The spiking neurons encode beneficial temporal facts and possess excessive anti-noise properties. However, the high-quality encoding of spatio-temporal complexity and also its training optimization of SNNs are restricted by means of the contemporary problem, this article proposes a novel hierarchical event-driven visual device to explore how information transmits and signifies in the retina the usage of biologically manageable mechanisms. This cognitive model is an augmented spiking-based framework consisting of the function learning capacity of convolutional neural networks (CNNs) with the cognition capability of SNNs. Furthermore, this visual device is modeled in a biological realism way with unsupervised learning rules and advanced spike firing rate encoding methods. We train and test them on some image datasets (Modified National Institute of Standards and Technology (MNIST), Canadian Institute for Advanced Research (CIFAR)10, and its noisy versions) to show that our mannequin can process greater vital data than present cognitive models. This article also proposes a novel quantization approach to make the proposed spiking-based model more efficient for neuromorphic hardware implementation. The outcomes show this joint CNN-SNN model can reap excessive focus accuracy and get more effective generalization ability.
Spiking Transformers for Event-based Single Object Tracking

Xin Yang,B. Dong,Baocai Yin,Jianchuan Ding,Felix Heide,Haiwei Zhang,Jiqing Zhang

DOI: https://doi.org/10.1109/CVPR52688.2022.00860

2022-06-01

Computer Vision and Pattern Recognition

Abstract:Event-based cameras bring a unique capability to tracking, being able to function in challenging real-world conditions as a direct result of their high temporal resolution and high dynamic range. These imagers capture events asynchronously that encode rich temporal and spatial information. However, effectively extracting this information from events remains an open challenge. In this work, we propose a spiking transformer network, STNet, for single object tracking. STNet dynamically extracts and fuses information from both temporal and spatial domains. In particular, the proposed architecture features a transformer module to provide global spatial information and a spiking neural network (SNN) module for extracting temporal cues. The spiking threshold of the SNN module is dynamically adjusted based on the statistical cues of the spatial information, which we find essential in providing robust SNN features. We fuse both feature branches dynamically with a novel cross-domain attention fusion algorithm. Extensive experiments on three event-based datasets, FE240hz, EED and VisEvent validate that the proposed STNet outperforms existing state-of-the-art methods in both tracking accuracy and speed with a significant margin. The code and pretrained models are at https://github.com/Jee-King/CVPR2022_STNet.

Computer Science
Toward High-Accuracy and Low-Latency Spiking Neural Networks With Two-Stage Optimization

Ziming Wang,Yuhao Zhang,Shuang Lian,Xiaoxin Cui,Rui Yan,Huajin Tang

DOI: https://doi.org/10.1109/TNNLS.2023.3337176

2023-12-15

Abstract:Spiking neural networks (SNNs) operating with asynchronous discrete events show higher energy efficiency with sparse computation. A popular approach for implementing deep SNNs is artificial neural network (ANN)-SNN conversion combining both efficient training of ANNs and efficient inference of SNNs. However, the accuracy loss is usually nonnegligible, especially under few time steps, which restricts the applications of SNN on latency-sensitive edge devices greatly. In this article, we first identify that such performance degradation stems from the misrepresentation of the negative or overflow residual membrane potential in SNNs. Inspired by this, we decompose the conversion error into three parts: quantization error, clipping error, and residual membrane potential representation error. With such insights, we propose a two-stage conversion algorithm to minimize those errors, respectively. In addition, we show that each stage achieves significant performance gains in a complementary manner. By evaluating on challenging datasets including CIFAR-10, CIFAR-100, and ImageNet, the proposed method demonstrates the state-of-the-art performance in terms of accuracy, latency, and energy preservation. Furthermore, our method is evaluated using a more challenging object detection task, revealing notable gains in regression performance under ultralow latency, when compared with existing spike-based detection algorithms. Codes will be available at: https://github.com/Windere/snn-cvt-dual-phase.
Corrigendum to "Improved STNNet, A benchmark for detection, tracking, and counting crowds using Drones" [MethodsX 13 (2024) 1-7/102820]

Mohd Nazeer,Kanhaiya Sharma,S Sathappan,Pulipati Srilatha,Arshad Ahmad Khan Mohammed

DOI: https://doi.org/10.1016/j.mex.2024.102870

2024-07-25

MethodsX

Abstract:[This corrects the article DOI: 10.1016/j.mex.2024.102820.].
Corrigendum: Multi-person feature fusion transfer learning-based convolutional neural network for SSVEP-based collaborative BCI

Penghai Li,Jianxian Su,Abdelkader Nasreddine Belkacem,Longlong Cheng,Chao Chen

DOI: https://doi.org/10.3389/fnins.2022.1024150

IF: 4.3

2022-10-26

Frontiers in Neuroscience

Abstract:[This corrects the article DOI: 10.3389/fnins.2022.971039.].

neurosciences
Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition

Yongqiang Cao,Yang Chen,Deepak Khosla

DOI: https://doi.org/10.1007/s11263-014-0788-3

IF: 13.369

2014-11-23

International Journal of Computer Vision

Abstract:Deep-learning neural networks such as convolutional neural network (CNN) have shown great potential as a solution for difficult vision problems, such as object recognition. Spiking neural networks (SNN)-based architectures have shown great potential as a solution for realizing ultra-low power consumption using spike-based neuromorphic hardware. This work describes a novel approach for converting a deep CNN into a SNN that enables mapping CNN to spike-based hardware architectures. Our approach first tailors the CNN architecture to fit the requirements of SNN, then trains the tailored CNN in the same way as one would with CNN, and finally applies the learned network weights to an SNN architecture derived from the tailored CNN. We evaluate the resulting SNN on publicly available Defense Advanced Research Projects Agency (DARPA) Neovision2 Tower and CIFAR-10 datasets and show similar object recognition accuracy as the original CNN. Our SNN implementation is amenable to direct mapping to spike-based neuromorphic hardware, such as the ones being developed under the DARPA SyNAPSE program. Our hardware mapping analysis suggests that SNN implementation on such spike-based hardware is two orders of magnitude more energy-efficient than the original CNN implementation on off-the-shelf FPGA-based hardware.

computer science, artificial intelligence
Spike-Based Motion Estimation for Object Tracking Through Bio-Inspired Unsupervised Learning

Yajing Zheng,Zhaofei Yu,Song Wang,Tiejun Huang

DOI: https://doi.org/10.1109/tip.2022.3228168

IF: 10.6

2022-12-24

IEEE Transactions on Image Processing

Abstract:Neuromorphic vision sensors, whose pixels output events/spikes asynchronously with a high temporal resolution according to the scene radiance change, are naturally appropriate for capturing high-speed motion in the scenes. However, how to utilize the events/spikes to smoothly track high-speed moving objects is still a challenging problem. Existing approaches either employ time-consuming iterative optimization, or require large amounts of labeled data to train the object detector. To this end, we propose a bio-inspired unsupervised learning framework, which takes advantage of the spatiotemporal information of events/spikes generated by neuromorphic vision sensors to capture the intrinsic motion patterns. Without off-line training, our models can filter the redundant signals with dynamic adaption module based on short-term plasticity, and extract the motion patterns with motion estimation module based on the spike-timing-dependent plasticity. Combined with the spatiotemporal and motion information of the filtered spike stream, the traditional DBSCAN clustering algorithm and Kalman filter can effectively track multiple targets in extreme scenes. We evaluate the proposed unsupervised framework for object detection and tracking tasks on synthetic data, publicly available event-based datasets, and spiking camera datasets. The experiment results show that the proposed model can robustly detect and smoothly track the moving targets on various challenging scenarios and outperforms state-of-the-art approaches.

computer science, artificial intelligence,engineering, electrical & electronic
Enabling energy-Efficient object detection with surrogate gradient descent in spiking neural networks

Jilong Luo,Shanlin Xiao,Yinsheng Chen,Zhiyi Yu

2023-09-07

Abstract:Spiking Neural Networks (SNNs) are a biologically plausible neural network model with significant advantages in both event-driven processing and spatio-temporal information processing, rendering SNNs an appealing choice for energyefficient object detection. However, the non-differentiability of the biological neuronal dynamics model presents a challenge during the training of SNNs. Furthermore, a suitable decoding strategy for object detection in SNNs is currently lacking. In this study, we introduce the Current Mean Decoding (CMD) method, which solves the regression problem to facilitate the training of deep SNNs for object detection tasks. Based on the gradient surrogate and CMD, we propose the SNN-YOLOv3 model for object detection. Our experiments demonstrate that SNN-YOLOv3 achieves a remarkable performance with an mAP of 61.87% on the PASCAL VOC dataset, requiring only 6 time steps. Compared to SpikingYOLO, we have managed to increase mAP by nearly 10% while reducing energy consumption by two orders of magnitude.

Computer Vision and Pattern Recognition,Artificial Intelligence
Correction: Spatiotemporal network coding of physiological mossy fiber inputs by the cerebellar granular layer

Shyam Kumar Sudhakar,Sungho Hong,Ivan Raikov,Rodrigo Publio,Claus Lang,Thomas Close,Daqing Guo,Mario Negrello,Erik de Schutter

DOI: https://doi.org/10.1371/journal.pcbi.1007472

2019-01-01

PLoS Computational Biology

Abstract:[This corrects the article DOI: 10.1371/journal.pcbi.1005754.].
Spiking Neural Network for Ultra-low-latency and High-accurate Object Detection

Jinye Qu,Zeyu Gao,Tielin Zhang,Yanfeng Lu,Huajin Tang,Hong Qiao

2023-06-27

Abstract:Spiking Neural Networks (SNNs) have garnered widespread interest for their energy efficiency and brain-inspired event-driven properties. While recent methods like Spiking-YOLO have expanded the SNNs to more challenging object detection tasks, they often suffer from high latency and low detection accuracy, making them difficult to deploy on latency sensitive mobile platforms. Furthermore, the conversion method from Artificial Neural Networks (ANNs) to SNNs is hard to maintain the complete structure of the ANNs, resulting in poor feature representation and high conversion errors. To address these challenges, we propose two methods: timesteps compression and spike-time-dependent integrated (STDI) coding. The former reduces the timesteps required in ANN-SNN conversion by compressing information, while the latter sets a time-varying threshold to expand the information holding capacity. We also present a SNN-based ultra-low latency and high accurate object detection model (SUHD) that achieves state-of-the-art performance on nontrivial datasets like PASCAL VOC and MS COCO, with about remarkable 750x fewer timesteps and 30% mean average precision (mAP) improvement, compared to the Spiking-YOLO on MS COCO datasets. To the best of our knowledge, SUHD is the deepest spike-based object detection model to date that achieves ultra low timesteps to complete the lossless conversion.

Computer Vision and Pattern Recognition,Neural and Evolutionary Computing
Object detection and tracking method based on spiking neuron network

Jing Zhou,Jun Xiong

DOI: https://doi.org/10.23919/CCC55666.2022.9901740

2022-07-25

Abstract:Aiming at the problems of large parameter capacity and heavy dependence on acceleration framework for complex target detection and tracking model, the conversion method between artificial neural network and impulse neural network is studied in this work to implement target detection and tracking task based on the spiking neural network. Firstly, the convolution modules Alexnet, vgg11-5c and vgg16-8c of the full convolution twin network Siamese-FC are properly adjusted and cut to adapt with the characteristics and limitations of impulse neurons, and then the Norm-SiamFC network model is constructed and trained. Finally, the migration parameters are adopted to construct a spiking network Spiking-SiamFC for object detection and tracking, which achieves good tracking effect and occupies less resources, thus the model can be widely utilized in edge devices with limited resource storage capacity.

Engineering,Computer Science
Corrigendum to "AFEX-Net: Adaptive feature extraction convolutional neural network for classifying computerized tomography images"

DOI: https://doi.org/10.1177/20552076241276671

2024-08-19

Abstract:[This corrects the article DOI: 10.1177/20552076241232882.].
Visual Object Tracking with Adaptive Structural Convolutional Network

Di Yuan,Xin Li,Zhenyu He,Qiao Liu,Shuwei Lu

DOI: https://doi.org/10.1016/j.knosys.2020.105554

IF: 8.139

2020-01-01

Knowledge-Based Systems

Abstract:Convolutional Neural Networks (CNN) have been demonstrated to achieve state-of-the-art performance in visual object tracking task. However, existing CNN-based trackers usually use holistic target samples to train their networks. Once the target undergoes complicated situations (e.g., occlusion, background clutter, and deformation), the tracking performance degrades badly. In this paper, we propose an adaptive structural convolutional filter model to enhance the robustness of deep regression trackers (named: ASCT). Specifically, we first design a mask set to generate local filters to capture local structures of the target. Meanwhile, we adopt an adaptive weighting fusion strategy for these local filters to adapt to the changes in the target appearance, which can enhance the robustness of the tracker effectively. Besides, we develop an end-to-end trainable network comprising feature extraction, decision making, and model updating modules for effective training. Extensive experimental results on large benchmark datasets demonstrate the effectiveness of the proposed ASCT tracker performs favorably against the state-of-the-art trackers.
Corrigendum: Deep Learning-Enabled Clinically Applicable CT Planbox for Stroke With High Accuracy and Repeatability

Yang Wang,Junkai Zhu,Jinli Zhao,Wenyi Li,Xin Zhang,Xiaolin Meng,Taige Chen,Ming Li,Meiping Ye,Renfang Hu,Shidan Dou,Huayin Hao,Xiaofen Zhao,Xiaoming Wu,Wei Hu,Cheng Li,Xiaole Fan,Liyun Jiang,Xiaofan Lu,Fangrong Yan

DOI: https://doi.org/10.3389/fneur.2022.921992

IF: 3.4

2022-05-09

Frontiers in Neurology

Abstract:[This corrects the article DOI: 10.3389/fneur.2022.755492.].

neurosciences,clinical neurology
Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection

Xinhao Luo,Man Yao,Yuhong Chou,Bo Xu,Guoqi Li

2024-08-05

Abstract:Brain-inspired Spiking Neural Networks (SNNs) have bio-plausibility and low-power advantages over Artificial Neural Networks (ANNs). Applications of SNNs are currently limited to simple classification tasks because of their poor performance. In this work, we focus on bridging the performance gap between ANNs and SNNs on object detection. Our design revolves around network architecture and spiking neuron. First, the overly complex module design causes spike degradation when the YOLO series is converted to the corresponding spiking version. We design a SpikeYOLO architecture to solve this problem by simplifying the vanilla YOLO and incorporating meta SNN blocks. Second, object detection is more sensitive to quantization errors in the conversion of membrane potentials into binary spikes by spiking neurons. To address this challenge, we design a new spiking neuron that activates Integer values during training while maintaining spike-driven by extending virtual timesteps during inference. The proposed method is validated on both static and neuromorphic object detection datasets. On the static COCO dataset, we obtain 66.2% mAP@50 and 48.9% mAP@50:95, which is +15.0% and +18.7% higher than the prior state-of-the-art SNN, respectively. On the neuromorphic Gen1 dataset, we achieve 67.2% mAP@50, which is +2.5% greater than the ANN with equivalent architecture, and the energy efficiency is improved by 5.7*. Code: <a class="link-external link-https" href="https://github.com/BICLab/SpikeYOLO" rel="external noopener nofollow">this https URL</a>

Artificial Intelligence
LSNT: A Lightweight Siamese Network Based Tracker

Xuezhen Dong,Zhangjin Huang,Lu Zou,Fangjun Wang,Zonghui Zhang

DOI: https://doi.org/10.1007/978-3-030-87361-5_52

2021-01-01

Abstract:Trackers based on the Siamese network have achieved remarkable advancements in accuracy and robustness, yet the trackers suffer from enormous computation and memory storage cost, which hinder their rapid inference, making it difficult to deploy in calculation-constrained and memory storage-constrained scenarios. Moreover, a general depthwise correlation attempts to process the entire feature map indiscriminately, which cannot extract more robust features. In this paper, we presented a Lightweight Siamese Network based Tracker (LSNT) to tackle the above issues. In the tracking head part, an extremely efficient module is adopted to simplify the network, so as to maintain a balance with the backbone network in parameter quantity. Furthermore, a novel Spatial-Channel Attention based Depthwise (SCAD) correlation is introduced to enhance the ability of the feature fusion module. Compared with the baseline tracker, LSNT has fewer parameters and calculations, but experiments conducted on the GOT-10k benchmark demonstrate that it achieves comparable performance in terms of accuracy and robustness, thus confirming its efficiency and effectiveness.

Corrigendum: SCTN: event-based object tracking with energy-efficient deep convolutional spiking neural networks

Corrigendum: A brain-inspired decision-making spiking neural network and its application in unmanned aerial vehicle

RETRACTED: Multiscale fast correlation filtering tracking algorithm based on a feature fusion model (Retracted article. See vol. 34, 2022)

Spiking SiamFC++: Deep Spiking Neural Network for Object Tracking

Spike Trains Encoding and Threshold Rescaling Method for Deep Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Spiking Transformers for Event-based Single Object Tracking

Toward High-Accuracy and Low-Latency Spiking Neural Networks With Two-Stage Optimization

Corrigendum to "Improved STNNet, A benchmark for detection, tracking, and counting crowds using Drones" [MethodsX 13 (2024) 1-7/102820]

Corrigendum: Multi-person feature fusion transfer learning-based convolutional neural network for SSVEP-based collaborative BCI

Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition

Spike-Based Motion Estimation for Object Tracking Through Bio-Inspired Unsupervised Learning

Enabling energy-Efficient object detection with surrogate gradient descent in spiking neural networks

Correction: Spatiotemporal network coding of physiological mossy fiber inputs by the cerebellar granular layer

Spiking Neural Network for Ultra-low-latency and High-accurate Object Detection

Object detection and tracking method based on spiking neuron network

Corrigendum to "AFEX-Net: Adaptive feature extraction convolutional neural network for classifying computerized tomography images"

Visual Object Tracking with Adaptive Structural Convolutional Network

Corrigendum: Deep Learning-Enabled Clinically Applicable CT Planbox for Stroke With High Accuracy and Repeatability

Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection

LSNT: A Lightweight Siamese Network Based Tracker