Abstract:Pedestrian Detection is the most critical module of an Autonomous Driving system. Although a camera is commonly used for this purpose, its quality degrades severely in low-light night time driving scenarios. On the other hand, the quality of a thermal camera image remains unaffected in similar conditions. This paper proposes an end-to-end multimodal fusion model for pedestrian detection using RGB and thermal images. Its novel spatio-contextual deep network architecture is capable of exploiting the multimodal input efficiently. It consists of two distinct deformable ResNeXt-50 encoders for feature extraction from the two modalities. Fusion of these two encoded features takes place inside a multimodal feature embedding module (MuFEm) consisting of several groups of a pair of Graph Attention Network and a feature fusion unit. The output of the last feature fusion unit of MuFEm is subsequently passed to two CRFs for their spatial refinement. Further enhancement of the features is achieved by applying channel-wise attention and extraction of contextual information with the help of four RNNs traversing in four different directions. Finally, these feature maps are used by a single-stage decoder to generate the bounding box of each pedestrian and the score map. We have performed extensive experiments of the proposed framework on three publicly available multimodal pedestrian detection benchmark datasets, namely KAIST, CVC-14, and UTokyo. The results on each of them improved the respective state-of-the-art performance. A short video giving an overview of this work along with its qualitative results can be seen at <a class="link-external link-https" href="https://youtu.be/FDJdSifuuCs" rel="external noopener nofollow">this https URL</a>. Our source code will be released upon publication of the paper.

Pedestrian Detection Based on Hand-crafted Features and Multi-layer Feature Fused-ResNet Model

A Novel Approach to Design the Fast Pedestrian Detection for Video Surveillance System

Towards Accurate Dense Pedestrian Detection Via Occlusion-Prediction Aware Label Assignment and Hierarchical-Nms.

Pedestrian Detection Using Multi-Channel Visual Feature Fusion by Learning Deep Quality Model.

MM_Fast_RCNN_ResNet: Construction of Multimodal Faster RCNN Inception and ResNet V2 for Pedestrian Tracking and detection

Multispectral Deep Neural Networks for Pedestrian Detection

Multi ROI and Multi Map Networks for Accurate and Efficient Pedestrian Detection

Transformer fusion and histogram layer multispectral pedestrian detection network

Pedestrian Detection Based on Region Proposal Fusion.

Multi-Scale Feature Pyramid Network: A Heavily Occluded Pedestrian Detection Network Based on ResNet

Robust multi-modal pedestrian detection using deep convolutional neural network with ensemble learning model

Deep convolutional neural networks for pedestrian detection

Autonomous pedestrian detection for crowd surveillance using deep learning framework

Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving

Two-Stage Pedestrian Detection Model Using a New Classification Head for Domain Generalization

Deep Neural Network Based Vehicle and Pedestrian Detection for Autonomous Driving: A Survey

Fast Pedestrian Detection with Attention-Enhanced Multi-Scale RPN and Soft-Cascaded Decision Trees

Multi-Grained Deep Feature Learning for Pedestrian Detection

Pedestrian Detection with Multi-View Convolution Fusion Algorithm

Deep Convolutional Neural Networks For Pedestrian Detection With Skip Pooling

SAM-RCNN: Scale-Aware Multi-Resolution Multi-Channel Pedestrian Detection