Abstract:Pedestrian Detection is the most critical module of an Autonomous Driving system. Although a camera is commonly used for this purpose, its quality degrades severely in low-light night time driving scenarios. On the other hand, the quality of a thermal camera image remains unaffected in similar conditions. This paper proposes an end-to-end multimodal fusion model for pedestrian detection using RGB and thermal images. Its novel spatio-contextual deep network architecture is capable of exploiting the multimodal input efficiently. It consists of two distinct deformable ResNeXt-50 encoders for feature extraction from the two modalities. Fusion of these two encoded features takes place inside a multimodal feature embedding module (MuFEm) consisting of several groups of a pair of Graph Attention Network and a feature fusion unit. The output of the last feature fusion unit of MuFEm is subsequently passed to two CRFs for their spatial refinement. Further enhancement of the features is achieved by applying channel-wise attention and extraction of contextual information with the help of four RNNs traversing in four different directions. Finally, these feature maps are used by a single-stage decoder to generate the bounding box of each pedestrian and the score map. We have performed extensive experiments of the proposed framework on three publicly available multimodal pedestrian detection benchmark datasets, namely KAIST, CVC-14, and UTokyo. The results on each of them improved the respective state-of-the-art performance. A short video giving an overview of this work along with its qualitative results can be seen at <a class="link-external link-https" href="https://youtu.be/FDJdSifuuCs" rel="external noopener nofollow">this https URL</a>. Our source code will be released upon publication of the paper.

Linear Support Tensor Machine: Pedestrian Detection in Thermal Infrared Images

A Fourier-Transform-Based Framework with Asymptotic Attention for Mobile Thermal InfraRed Object Detection

Pedestrian Detection Based on Center, Temperature, Scale and Ratio Prediction in Thermal Imagery

A Novel Approach to Design the Fast Pedestrian Detection for Video Surveillance System

An effective approach to pedestrian detection in thermal imagery.

Pedestrian Detection in Complex Scene Using Full Binary Tree Classifiers Based on Locally Assembled Binary Haar-Like Features

Robust Thermal Infrared Pedestrian Detection By Associating Visible Pedestrian Knowledge

A Robust Thermal Infrared Vehicle and Pedestrian Detection Method in Complex Scenes

Effective Pedestrian Detection Using Center-symmetric Local Binary/Trinary Patterns

Pedestrian Detection Based on Bidirectional Local Template Patterns.

Shape-based Pedestrian Detection in Infrared Images

Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving

A new thermal infrared and visible spectrum images-based pedestrian detection system

Bidirectional Local Template Patterns: an Effective and Discriminative Feature for Pedestrian Detection

A Novel Method for Tracking Pedestrians from Real-Time Video.

Robust pedestrian detection in thermal infrared imagery using a shape distribution histogram feature and modified sparse representation classification

Learning Cross-Modal Deep Representations for Robust Pedestrian Detection

Thermal imaging pedestrian detection algorithm based on attention guidance and local cross-level network

Pedestrian Detection with Lidar Point Clouds Based on Single Template Matching

Robust pedestrian detection in thermal imagery using synthesized images

Steerable Second Order Intensity Features For Pedestrian Detection