Abstract:We present Parametric Piecewise Linear Networks (PPLNs) for temporal vision inference. Motivated by the neuromorphic principles that regulate biological neural behaviors, PPLNs are ideal for processing data captured by event cameras, which are built to simulate neural activities in the human retina. We discuss how to represent the membrane potential of an artificial neuron by a parametric piecewise linear function with learnable coefficients. This design echoes the idea of building deep models from learnable parametric functions recently popularized by Kolmogorov-Arnold Networks (KANs). Experiments demonstrate the state-of-the-art performance of PPLNs in event-based and image-based vision applications, including steering prediction, human pose estimation, and motion deblurring. The source code of our implementation is available at <a class="link-external link-https" href="https://github.com/chensong1995/PPLN" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve event - based visual reasoning problems, especially how to perform efficient spatio - temporal modeling using the data captured by event cameras. Specifically, the paper proposes a new neural network architecture - Parametric Piecewise Linear Networks (PPLNs) - to process the time - series data generated by event cameras. #### Main problems and background 1. **Advantages and challenges of event cameras**: - Event cameras are bionic sensors that can capture environmental changes in a high - dynamic - range, low - latency, and low - power manner. - Compared with traditional cameras, the data generated by event cameras is represented as a series of discrete events, and each event contains pixel coordinates, a timestamp, and the polarity of luminance change. - Although event cameras have many advantages, their non - traditional data representation poses challenges to existing computer vision algorithms. 2. **Limitations of existing methods**: - Existing event - based visual algorithms usually rely on multi - layer perceptrons or multi - convolution operations, and these methods fail to fully utilize the temporal characteristics of event data. - These methods perform poorly when dealing with complex tasks such as motion deblurring and human pose estimation. 3. **Motivation for the design of PPLNs**: - PPLNs aim to process event data by imitating the behavior of biological neurons. Specifically, PPLNs model the membrane potential of neurons as a piecewise linear function, thereby better capturing time information. - This design draws on the ideas of Kolmogorov - Arnold Networks (KANs) and the Leaky Integrate - and - Fire model, while also incorporating the characteristics of the event generation model. #### Solutions - **Parametric Piecewise Linear Networks (PPLNs)**: - PPLNs construct piecewise linear functions by learning the linear coefficients of the input data to simulate the change of the membrane potential of neurons over time. - This network can evaluate the neuron output at any time point and improve numerical stability through smoothing and normalization operations. - **Application areas**: - The paper demonstrates the superior performance of PPLNs in multiple tasks, including: - **Motion deblurring**: Recovering clear videos from blurred images. - **Steering angle prediction**: Predicting the steering angle of a vehicle based on dash - cam data. - **Human pose estimation**: Estimating 3D human poses from binocular event camera data. #### Experimental results - PPLNs achieved significant performance improvements in all tested tasks. In particular, in the motion deblurring task, compared with the best existing method, PPLN improved by 5.6% in MSE, 0.372 dB in PSNR, and 4.9% in SSIM respectively. - In the steering angle prediction and human pose estimation tasks, PPLNs also performed well, with improvements of 30.8% and 11.1% respectively. In conclusion, by introducing PPLNs, this paper provides an efficient and general method for handling event - based visual tasks, demonstrating the great potential of bionic neural networks in spatio - temporal modeling.

PPLNs: Parametric Piecewise Linear Networks for Event-Based Temporal Modeling and Beyond

Piecewise Linear Neural Networks and Deep Learning

Neural Point Process for Learning Spatiotemporal Event Dynamics

TENNs-PLEIADES: Building Temporal Kernels with Orthogonal Polynomials

Predictive Coding Networks for Temporal Prediction

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception.

Learning Neural Point Processes with Latent Graphs

EDeNN: Event Decay Neural Networks for low latency vision

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras

Event enhanced Passive Non-line-of-sight imaging for moving objects with Physical embedding

Deep LPPLS: Forecasting of temporal critical points in natural, engineering and financial systems

RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network

A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Extended Poisson Gaussian-Process Latent Variable Model for Unsupervised Neural Decoding

Spatio-Temporal Adaptation in the Unsupervised Development of Networked Visual Neurons

Predictive Encoding of Contextual Relationships for Perceptual Inference, Interpolation and Prediction.

Shallow neural networks trained to detect collisions recover features of visual loom-selective neurons

PLM-Net: Perception Latency Mitigation Network for Vision-Based Lateral Control of Autonomous Vehicles