Abstract:Recently, convolutional neural networks (CNNs) are the leading defacto method for crowd counting. However, when dealing with video datasets, CNN-based methods still process each video frame independently, thus ignoring the powerful temporal information between consecutive frames. In this work, we propose a novel architecture termed as "temporal channel-aware" (TCA) block, which achieves the capability of exploiting the temporal interdependencies among video sequences. Specifically, we incorporate 3D convolution kernels to encode local spatio-temporal features. Furthermore, the global contextual information is encoded into modulation weights which adaptively recalibrate channel-aware feature responses. With the local and global context combined, the proposed block enhances the discriminative ability of the feature representations and contributes to more precise results in diverse scenes. By stacking TCA blocks together, we obtain the deep trainable architecture called enhanced 3D convolutional networks (E3D). The experiments on three benchmark datasets show that the proposed method delivers state-of-the-art performance. To verify the generality, an extended experiment is conducted on a vehicle dataset TRANCOS and our approach beats previous methods by large margins.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is that when performing crowd counting in video datasets, the existing methods based on Convolutional Neural Networks (CNN) usually process each video frame independently, ignoring the strong temporal information between consecutive frames. Specifically, the paper proposes a new architecture - Enhanced 3D Convolutional Networks (E3D) to utilize the spatio - temporal information in video sequences, thereby improving the accuracy of crowd counting. ### Main problems: 1. **Ignoring temporal information**: When processing video data, existing methods still regard each frame as an independent static image, ignoring the temporal correlation between frames. 2. **Limitation of local receptive fields**: Traditional CNN methods mainly rely on local receptive fields to extract features, which makes it difficult for the model to capture the global scene context. 3. **High computational complexity**: Some methods need to train additional complex network structures in order to introduce global context information, resulting in excessive computational costs. ### Solutions: To solve the above problems, the author proposes the following innovations: - **Temporal Channel - Aware (TCA) Block**: A new TCA block is designed, which combines local and global spatio - temporal information. By introducing 3D convolution kernels to encode local spatio - temporal features and converting global context information into modulation weights, the channel - aware feature responses are adaptively recalibrated. - **3D Convolution**: Use 3D convolution kernels to capture the spatio - temporal correlations between consecutive frames in the video, especially enhancing the representational ability in the temporal dimension. - **E3D Network**: By stacking multiple TCA blocks, a deep - trainable network architecture E3D is constructed, which can better handle the spatio - temporal information in video data. ### Experimental results: The experiments were verified on three benchmark datasets (WorldExpo’10, UCSD, and MALL) and a vehicle dataset TRANCOS. The results show that E3D has achieved state - of - the - art performance on these datasets. In particular, when processing video data, E3D significantly outperforms other methods, proving its effectiveness in utilizing temporal information. ### Summary: By introducing 3D convolution and TCA blocks, this paper successfully solves the problem that existing methods ignore temporal information when processing video data, and improves the accuracy of crowd counting. In addition, the paper also demonstrates the versatility of this model in different application scenarios, such as vehicle counting.

Enhanced 3D convolutional networks for crowd counting

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Multi-branch Progressive Embedding Network for Crowd Counting

ST-CNN: Spatial-Temporal Convolutional Neural Network for Crowd Counting in Videos.

Concise Convolutional Neural Network for Crowd Counting

Crowd Counting Via Enhanced Feature Channel Convolutional Neural Network

LEVERAGE MULTI-SCALE DILATED CONVOLUTIONAL NEURAL NETWORK WITH GLOBAL ATTENTION FEATURE FUSION FOR CROWD COUNTING

TFA-CNN: an efficient method for dealing with crowding and noise problems in crowd counting

An Adaptive Multi-Scale Network Based on Depth Information for Crowd Counting

Attend to Count: Crowd Counting with Adaptive Capacity Multi-Scale CNNs.

End to End Multi-Scale Convolutional Neural Network for Crowd Counting.

PointerNet - Spatiotemporal Modeling for Crowd Counting in Videos.

Crowd Counting Using Deep Recurrent Spatial-Aware Network.

Crowd counting via region based multi-channel convolution neural network

Multi-scale Dilated Convolution of Feature Fusion Network for Crowd Counting

HADF-Crowd: A Hierarchical Attention-Based Dense Feature Extraction Network for Single-Image Crowd Counting

Counting Congested Crowds under Wild Conditions with a Multi-Task Inception Network

Deep People Counting in Extremely Dense Crowds

Crowd Counting From A Still Image Using Multi-Scale Fully Convolutional Network With Adaptive Human-Shaped Kernel

$$\hbox {DA}^2$$Net: a dual attention-aware network for robust crowd counting

Spatiotemporal Dilated Convolution with Uncertain Matching for Video-based Crowd Estimation