Abstract:Large language models (LLMs) have shown their capabilities in understanding contextual and semantic information regarding knowledge of instance appearances. In this paper, we introduce a novel approach to utilize the strengths of LLMs in understanding contextual appearance variations and to leverage this knowledge into a vision model (here, pedestrian detection). While pedestrian detection is considered one of the crucial tasks directly related to our safety (e.g., intelligent driving systems), it is challenging because of varying appearances and poses in diverse scenes. Therefore, we propose to formulate language-derived appearance elements and incorporate them with visual cues in pedestrian detection. To this end, we establish a description corpus that includes numerous narratives describing various appearances of pedestrians and other instances. By feeding them through an LLM, we extract appearance knowledge sets that contain the representations of appearance variations. Subsequently, we perform a task-prompting process to obtain appearance elements which are guided representative appearance knowledge relevant to a downstream pedestrian detection task. The obtained knowledge elements are adaptable to various detection frameworks, so that we can provide plentiful appearance information by integrating the language-derived appearance elements with visual cues within a detector. Through comprehensive experiments with various pedestrian detectors, we verify the adaptability and effectiveness of our method showing noticeable performance gains and achieving state-of-the-art detection performance on two public pedestrian detection benchmarks (i.e., CrowdHuman and WiderPedestrian).

Part-Based Pedestrian Detection Using Grammar Model And Abm-Hog Features

Discriminative Hierarchical Part-Based Models for Human Parsing and Action Recognition.

Pedestrian Detection Via Part-Based Topology Model

Pedestrian Detection in Complex Scene Using Full Binary Tree Classifiers Based on Locally Assembled Binary Haar-Like Features

A Pedestrian-Detection Method Based on Heterogeneous Features and Ensemble of Multi-View–Pose Parts

Enhanced Deformable Part Model for Pedestrian Detection Via Joint State Inference

Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection

Human Parsing Using Stochastic And-or Grammars and Rich Appearances

Representation and Analysis of Pedestrian Crossing States Based on Video Tracking

Pedestrian detection based on modified dynamic background using gaussian mixture models and hog-svm detection

Pedestrian Detection Method Based on Part Detector and Substructure Assemble

Pedestrian Detection by Learning a Mixture Mask Model and Its Implementation

Analysis and Prediction of Pedestrians' Violation Behavior at the Intersection Based on a Markov Chain

Double-pedestrian deformable part model and its application in pedestrian detection

Efficient Pedestrian Detection via Rectangular Features Based on a Statistical Shape Model

Pedestrian Detection for Autonomous Vehicles Using Virtual-to-Real Augmentation

BR-GAN: A Pedestrian Trajectory Prediction Model Combined With Behavior Recognition

Pedestrian Detection Using a Mixture Mask Model

Pedestrian Motion State Estimation From 2D Pose

A Monocular Vision Based Pedestrian Detection System for Intelligent Vehicles

MGD-GAN: Text-to-Pedestrian Generation Through Multi-grained Discrimination