DCT-Cam for Single-shot Action Recognition

Yu Liang,Honghao Huang,Jingwei Li,Xiaowen Dong,Minghua Chen,Sigang Yang,Hongwei Chen

DOI: https://doi.org/10.1364/cleo_si.2022.sm1d.3

2022-01-01

Abstract:We propose a DCT-Cam to capture the video spectrum directly from the scene in a single shot, which can reduce 75% data requirement and achieve the same accuracy for the action recognition.

What problem does this paper attempt to address?

DoCam: Depth Sensing with an Optical Image Stabilization Supported RGB Camera.

Hao Pan,Feitong Tan,Yi-Chao Chen,Gaoang Huang,Qingyang Li,Wenhao Li,Guangtao Xue,Lili Qiu,Xiaoyu Ji

DOI: https://doi.org/10.1145/3495243.3560523

2022-01-01

Abstract:Optical image stabilizers (OIS) are widely used in digital cameras to counteract motion blur caused by camera shakes in capturing videos and photos. In this paper, we sought to expand the applicability of the lens-shift OIS technology for metric depth estimation, i.e., let a RGB camera to achieve the similar function of a time-of-flight (ToF) camera. Instead of having to move the entire camera for depth estimation, we propose DoCam, which controls the lens motion in the OIS module to achieve 3D reconstruction. After controlling the lens motion by altering the MEMS gyroscopes readings through acoustic injection, we improve the traditional bundle adjustment algorithm by establishing additional constraints from the linearity of the lens control model for high-precision camera pose estimation. Then, we elaborate a dense depth reconstruction algorithm to compute depth maps at real-world scale from multiple captures with micro lens motion (i.e., <= 3 mm). Extensive experiments demonstrate that our proposed DoCam can enable a 2D color camera to estimate high-accuracy depth information of the captured scene by means of controlling lens motion in the OIS. DoCam is suitable for a variety of applications that require depth information of the scenes, especially when only a single color camera is available and located at a fixed position.
Action Recognition Based on Discrete Cosine Transform by Optical Pixel-Wise Encoding

Yu Liang,Honghao Huang,Jingwei Li,Xiaowen Dong,Minghua Chen,Sigang Yang,Hongwei Chen

DOI: https://doi.org/10.1063/5.0109807

IF: 5.6

2022-01-01

APL Photonics

Abstract:The framework provides a novel pipeline for action recognition. The action recognition task classifies the action label of the scene. High-speed cameras are commonly used to generate high frame-rate videos for capturing sufficient motion information. However, the data volume would be the bottleneck of the system. With the insight that the discrete cosine transform (DCT) of video signals reveals the motion information remarkably, instead of obtaining video data as with traditional cameras, the proposed method directly captures a DCT spectrum of video in a single shot through optical pixel-wise encoding. Considering that video signals are sparsely distributed in the DCT domain, a learning-based frequency selector is designed for pruning the trivial frequency channels of the spectrum. An opto-electronic neural network is designed for action recognition from a single coded spectrum. The optical encoder generates the DCT spectrum, and the rest of the network jointly optimizes the frequency selector and classification model simultaneously. Compared to conventional video-based action recognition methods, the proposed method achieves higher accuracy with less data, less communication bandwidth, and less computational burden. Both simulations and experiments demonstrate that the proposed method has superior action recognition performance. To the best of our knowledge, this is the first work investigating action recognition in the DCT domain.
A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Huafeng Wang,Tao Xia,Hanlin Li,Xianfeng Gu,Weifeng Lv,Yuehai Wang

DOI: https://doi.org/10.3390/math9243226

IF: 2.4

2021-01-01

Mathematics

Abstract:A very challenging task for action recognition concerns how to effectively extract and utilize the temporal and spatial information of video (especially temporal information). To date, many researchers have proposed various spatial-temporal convolution structures. Despite their success, most models are limited in further performance especially on those datasets that are highly time-dependent due to their failure to identify the fusion relationship between the spatial and temporal features inside the convolution channel. In this paper, we proposed a lightweight and efficient spatial-temporal extractor, denoted as Channel-Wise Spatial-Temporal Aggregation block (CSTA block), which could be flexibly plugged in existing 2D CNNs (denoted by CSTANet). The CSTA Block utilizes two branches to model spatial-temporal information separately. In temporal branch, It is equipped with a Motion Attention Module (MA), which is used to enhance the motion regions in a given video. Then, we introduced a Spatial-Temporal Channel Attention (STCA) module, which could aggregate spatial-temporal features of each block channel-wisely in a self-adaptive and trainable way. The final experimental results demonstrate that the proposed CSTANet achieved the state-of-the-art results on EGTEA Gaze++ and Diving48 datasets, and obtained competitive results on Something-Something V1&V2 at the less computational cost.
Compressed Video Action Recognition with Dual-Stream and Dual-Modal Transformer

Yuting Mou,Xinghao Jiang,Ke Xu,Tanfeng Sun,Zepeng Wang

DOI: https://doi.org/10.1109/tcsvt.2023.3319140

IF: 5.859

2024-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Compressed video action recognition offers the advantage of reducing decoding and inference time compared to the RGB domain. However, the compressed domain poses unique challenges with different types of frames (I-frames and P-frames). I-frames consistent with RGB are rich in frame information, but the redundant information may interfere with the recognition task. There are two modalities in P-frames, residual (R) and motion vector (MV). Although with less information, they can reflect the motion cue. To address these challenges and leverage the independent information from different frames and modalities, we propose a novel approach called Dual-Stream and Dual-Modal Transformer (DSDMT). Our approach consists of two streams: 1) The short-span P-frames stream contains temporal information. We propose the Dual-Modal Attention Module (DAM) to mine different modal variability in P-frames and complement the orthogonal feature vector. Besides, considering the sparsity of P-frames, we extract action features with Frame-level Patch Embedding (FPE) to avoid redundant computation. 2) The long-span I-frames stream extracts the global context feature of the entire video, including content and scene information. By fusing the global video context and local key-frame features, our model represents the action feature in terms of fine-grained and coarse-grained. We evaluated our proposed DSDMT on three public benchmarks with different scales: HMDB-51, UCF-101, and Kinetics-400. Ours achieve better performance with fewer Flops and lower latency. Our analysis shows that the independence and complements of the I-frames and P-frames extracted from the compressed video stream play a crucial role in action recognition.
Action Recognition for Depth Video using Multi-view Dynamic Images

Yang Xiao,Jun Chen,Yancheng Wang,Zhiguo Cao,Joey Tianyi Zhou,Xiang Bai

DOI: https://doi.org/10.48550/arXiv.1806.11269

2018-12-28

Abstract:Dynamic imaging is a recently proposed action description paradigm for simultaneously capturing motion and temporal evolution information, particularly in the context of deep convolutional neural networks (CNNs). Compared with optical flow for motion characterization, dynamic imaging exhibits superior efficiency and compactness. Inspired by the success of dynamic imaging in RGB video, this study extends it to the depth domain. To better exploit three-dimensional (3D) characteristics, multi-view dynamic images are proposed. In particular, the raw depth video is densely projected with respect to different virtual imaging viewpoints by rotating the virtual camera within the 3D space. Subsequently, dynamic images are extracted from the obtained multi-view depth videos and multi-view dynamic images are thus constructed from these images. Accordingly, more view-tolerant visual cues can be involved. A novel CNN model is then proposed to perform feature learning on multi-view dynamic images. Particularly, the dynamic images from different views share the same convolutional layers but correspond to different fully connected layers. This is aimed at enhancing the tuning effectiveness on shallow convolutional layers by alleviating the gradient vanishing problem. Moreover, as the spatial occurrence variation of the actions may impair the CNN, an action proposal approach is also put forth. In experiments, the proposed approach can achieve state-of-the-art performance on three challenging datasets.

Computer Vision and Pattern Recognition
RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks

Haokui Zhang,Ying Li,Peng Wang,Yu Liu,Chunhua Shen

2018-01-01

Abstract:Different from RGB videos, depth data in RGB-D videos provide key complementary information for tristimulus visual data which potentially could achieve accuracy improvement for action recognition. However, most of the existing action recognition models solely using RGB videos limit the performance capacity. Additionally, the state-of-the-art action recognition models, namely 3D convolutional neural networks (3D-CNNs) contain tremendous parameters suffering from computational inefficiency. In this paper, we propose a series of 3D light-weight architectures for action recognition based on RGB-D data. Compared with conventional 3D-CNN models, the proposed light-weight 3D-CNNs have considerably less parameters involving lower computation cost, while it results in favorable recognition performance. Experimental results on two public benchmark datasets show that our models can approximate or outperform the state-of-the-art approaches. Specifically, on the RGB+D-NTU (NTU) dataset, we achieve 93.2 measurement, and on the Northwestern-UCLA Multiview Action 3D (N-UCLA) dataset, we achieve 95.5
Full-resolution and full-dynamic-range coded aperture compressive temporal imaging

Ping Wang,Lishun Wang,Mu Qiao,Xin Yuan

DOI: https://doi.org/10.1364/OL.499735

2023-09-15

Abstract:Coded aperture compressive temporal imaging (CACTI) aims to capture a sequence of video frames in a single shot, using an off-the-shelf 2D sensor. This approach effectively increases the frame rate of the sensor while reducing data throughput requirements. However, previous CACTI systems have encountered challenges such as limited spatial resolution and a narrow dynamic range, primarily resulting from suboptimal optical modulation and sampling schemes. In this Letter, we present a highly efficient CACTI system that addresses these challenges by employing precise one-to-one pixel mapping between the sensor and modulator, while using structural gray scale masks instead of binary masks. Moreover, we develop a hybrid convolutional-Transformer deep network for accurate reconstruction of the captured frames. Both simulated and real data experiments demonstrate the superiority of our proposed system over previous approaches, exhibiting significant improvements in terms of spatial resolution and dynamic range.
3D-TDC: A 3D temporal dilation convolution framework for video action recognition

Yue Ming,Fan Feng,Chao Li,Jing-Hao Xue

DOI: https://doi.org/10.1016/j.neucom.2021.03.120

IF: 6

2021-08-01

Neurocomputing

Abstract:<p>Video action recognition is a vital area of computer vision. By adding temporal dimension into convolution structure, 3D convolution neural network owns the capacity to extract spatio-temporal features from videos. However, due to computing constraints, it is hard to input the whole video into the convolution network at one time, resulting in a limited temporal receptive field of the network. To address this issue, we propose a novel 3D temporal dilation convolution (3D-TDC) framework, to extract spatio-temporal features of actions from videos. First, we deploy the 3D temporal dilation convolution as the shallow temporal compression layer, enabling an effective capture of spatio-temporal information in a larger time domain with the reduced computational load. Then, an action recognition framework is constructed by integrating two networks with different temporal receptive fields to balance the long-short time difference. We conduct extensive experiments on three widely-used public datasets (UCF-101, HMDB-51, and Kinetics-400) for performance evaluation, and the experimental results demonstrate the effectiveness of our proposed framework in video action recognition with low computational load.</p>

computer science, artificial intelligence
Action Recognition and Benchmark Using Event Cameras.

Yue Gao,Jiaxuan Lu,Siqi Li,Nan Ma,Shaoyi Du,Yipeng Li,Qionghai Dai

DOI: https://doi.org/10.1109/tpami.2023.3300741

IF: 23.6

2023-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Recent years have witnessed remarkable achievements in video-based action recognition. Apart from traditional frame-based cameras, event cameras are bio-inspired vision sensors that only record pixel-wise brightness changes rather than the brightness value. However, little effort has been made in event-based action recognition, and large-scale public datasets are also nearly unavailable. In this paper, we propose an event-based action recognition framework called EV-ACT. The Learnable Multi-Fused Representation (LMFR) is first proposed to integrate multiple event information in a learnable manner. The LMFR with dual temporal granularity is fed into the event-based slow-fast network for the fusion of appearance and motion features. A spatial-temporal attention mechanism is introduced to further enhance the learning capability of action recognition. To prompt research in this direction, we have collected the largest event-based action recognition benchmark named THUE-ACT-50 and the accompanying THUE-ACT-50-CHL dataset under challenging environments, including a total of over 12,830 recordings from 50 action categories, which is over 4 times the size of the previous largest dataset. Experimental results show that our proposed framework could achieve improvements of over 14.5%, 7.6%, 11.2%, and 7.4% compared to previous works on four benchmarks. We have also deployed our proposed EV-ACT framework on a mobile platform to validate its practicality and efficiency.
Content-Aware Attention Network For Action Recognition

Zi-yi Liu,Le Wang,Nanning Zheng

DOI: https://doi.org/10.1007/978-3-319-92007-8_10

2018-01-01

Abstract:This paper presents a content-aware attention network (CatNet) for action recognition task, which can leverage attention mechanism to aggregate frame-level features into a compact video-level representation. Unlike most previous methods that consider every video frame equally, our CatNet contains an attention module which can adaptively emphasize the representative frames, and thus can benefit the action recognition task. Moreover, the CatNet can take an action video with arbitrary length yet produce a compact video representation with fixed length. The attention module consists of two cascaded blocks, an adaptive attention weighting block and a content-aware weighting block. The experiments are carried on two challenging video action datasets, i.e., the UCF-101 dataset and HMDB-51 dataset. Our method achieves significantly improvements on both datasets compared with existing methods. The results show that our proposed CatNet is able to focus on the representative frames corresponding to a specific action category, and meanwhile significantly improve the recognition performance.
D3D: Dual 3-D Convolutional Network for Real-Time Action Recognition

Shengqin Jiang,Yuankai Qi,Haokui Zhang,Zongwen Bai,Xiaobo Lu,Peng Wang

DOI: https://doi.org/10.1109/tii.2020.3018487

IF: 12.3

2021-07-01

IEEE Transactions on Industrial Informatics

Abstract:Three-dimensional convolutional neural networks (3D CNNs) have been explored to learn spatio-temporal information for video-based human action recognition. Expensive computational cost and memory demand resulted from standard 3D CNNs, however, hinder their application in practical scenarios. In this article, we address the aforementioned limitations by proposing a novel dual 3-D convolutional network (D3DNet) with two complementary lightweight branches. A coarse branch maintains large temporal receptive field by a fast temporal downsampling strategy and simulates the expensive 3-D convolutions using a combination of more efficient spatial convolutions and temporal convolutions. Meanwhile, a fine branch progressively downsamples the video in the temporal domain and adopts 3-D convolutional units with reduced channel capacities to capture multiresolution spatio-temporal information. Instead of learning these two branches independently, a shallow spatiotemporal downsampling module is shared for these two branches for efficient low-level feature learning. Besides, lateral connections are learned to effectively fuse the information from the two branches at multiple stages. The proposed network makes good balance between inference speed and action recognition performance. Based on RGB information only, it achieves competing performance on five popular video-based action recognition datasets, with inference speed of 3200FPS on a single NVIDIA GTX 2080Ti card.

automation & control systems,computer science, interdisciplinary applications,engineering, industrial
Extreme Low-Resolution Action Recognition with Confident Spatial-Temporal Attention Transfer

Yucai Bai,Qin Zou,Xieyuanli Chen,Lingxi Li,Zhengming Ding,Long Chen

DOI: https://doi.org/10.1007/s11263-023-01771-4

IF: 13.369

2023-03-09

International Journal of Computer Vision

Abstract:Action recognition on extreme low-resolution videos, e.g., a resolution of pixels, plays a vital role in far-view surveillance and privacy-preserving multimedia analysis. As low-resolution videos often only contain limited information, it is difficult for us to perform action recognition in them. Given the fact that one same action may be represented by videos in both high resolution (HR) and extreme low resolution (eLR), it is worth studying to utilize the relevant HR data to improve the eLR action recognition. In this work, we propose a novel Confident Spatial-Temporal Attention Transfer (CSTAT) for eLR action recognition. CSTAT acquires information from HR data by reducing the attention differences with a transfer-learning strategy. Besides, the confidence of the supervisory signal is also taken into consideration for a more reliable transferring process. Experimental results demonstrate that, the proposed method can effectively improve the accuracy of eLR action recognition and achieve state-of-the-art performances on HMDB51, Kinects-400, and Something-Something v2.

computer science, artificial intelligence
Action Recognition with Multi-Scale Trajectory-Pooled 3D Convolutional Descriptors

Xiusheng Lu,Hongxun Yao,Sicheng Zhao,Xiaoshuai Sun,Shengping Zhang

DOI: https://doi.org/10.1007/s11042-017-5251-3

IF: 2.577

2017-01-01

Multimedia Tools and Applications

Abstract:Hand-crafted and learning-based features are two main types of video representations in the field of video understanding. How to integrate their merits to design good descriptors has been the research hotspot recently. Motivated by TDD (Wang et al. 2015 ), we combine trajectory pooling method and 3D ConvNets (Tran et al. 2015 ) and put forward a novel multi-scale trajectory-pooled 3D convolutional descriptor (MTC3D) for action recognition in this paper. Specifically, we calculate multi-scale dense trajectories from the input video and perform trajectory pooling on feature maps of 3D CNN. The proposed descriptor has two advantages: 3D CNN has the ability to extract high-level semantic information from videos and multi-scale trajectory pooling method utilizes the temporal information of videos subtly. The experiments on the datasets of HMDB51 and UCF101 demonstrate that the proposed descriptor achieves state-of-the-art results.
DTCM: Joint Optimization of Dark Enhancement and Action Recognition in Videos

Zhigang Tu,Yuanzhong Liu,Yan Zhang,Qizi Mu,Junsong Yuan

DOI: https://doi.org/10.1109/tip.2023.3286254

IF: 10.6

2023-06-27

IEEE Transactions on Image Processing

Abstract:Recognizing human actions in dark videos is a useful yet challenging visual task in reality. Existing augmentation-based methods separate action recognition and dark enhancement in a two-stage pipeline, which leads to inconsistently learning of temporal representation for action recognition. To address this issue, we propose a novel end-to-end framework termed Dark Temporal Consistency Model (DTCM), which is able to jointly optimize dark enhancement and action recognition, and force the temporal consistency to guide downstream dark feature learning. Specifically, DTCM cascades the action classification head with the dark augmentation network to perform dark video action recognition in a one-stage pipeline. Our explored spatio-temporal consistency loss, which utilizes the RGB-Difference of dark video frames to encourage temporal coherence of the enhanced video frames, is effective for boosting spatio-temporal representation learning. Extensive experiments demonstrated that our DTCM has remarkable performance: 1) Competitive accuracy, which outperforms the state-of-the-arts on the ARID dataset by 2.32% and the UAVHuman-Fisheye dataset by 4.19% in accuracy, respectively; 2) High efficiency, which surpasses the current most advanced method (Chen et al., 2021) with only 6.4% GFLOPs and 71.3% number of parameters; 3) Strong generalization, which can be used in various action recognition methods (e.g., TSM, I3D, 3D-ResNext-101, Video-Swin) to promote their performance significantly.

computer science, artificial intelligence,engineering, electrical & electronic
DA-CCD: A novel action representation by Deep Architecture of local depth feature

Yi Liu,Lei Qin,Zhongwei Cheng,Yanhao Zhang,Weigang Zhang,Qingming Huang

DOI: https://doi.org/10.1109/ICIP.2014.7025167

2014-01-01

ICIP

Abstract:With the widespread use of depth sensors, it is crucial to provide an effective and efficient solution for human action analysis applications upon the informative depth data. In this paper, we present a generic framework of modeling the human action by deep architecture enhanced local features with depth data. To introduce robust higher-level representations, we augment the adaptive and scalable local depth features in a deep feature learning manner. Specifically, a Deep Architecture of Comparative Coding Descriptor (DA-CCD) is proposed to represent the depth action data. Our approach obtains consistently superior recognition precisions on view specific/non-specific scenarios compared with other leading action representations of depth data on the Huawei/3DLife Dataset.
A Cascaded Timestamp-Free Event Camera Image Compression Method for Gesture Recognition

Lindong Wu,Xuefei Lv,Shaoning Jin,Chaoyi Ban,Zongwei Wang,Yuan Wang,Yimao Cai

DOI: https://doi.org/10.1109/isie54533.2024.10595702

2024-01-01

Abstract:Event cameras have drawn great attention due to various advantages compared with frame-based cameras and have been used in many scenarios including gesture recognition. However, due to the high temporal resolution and the existence of timestamp in each event, the huge data amount still poses challenges for real-time processing in household and industrial scenes. To settle this problem, in this paper, we propose a cascaded timestamp-free compression (CTFC) method for reducing redundant data of event camera and demonstrate the effectiveness based on DVS Gesture dataset. The method includes two parts, designed for getting rid of the timestamps and filtering background (BA) noises, respectively. In the first part, a novel event number-fixed compression (ENFC) scheme is presented to remove all of the timestamps directly. The results indicate that 68% of data are omitted before transmission. In the second part, a spatial-correlation compression (SCC) scheme is proposed to further filter information-independent BA noises. The results show that up to 37.6% of the residual events are further removed. Moreover, we construct a spiking neural network (SNN) for gesture recognition to monitor the impact of CTFC method on event camera image quality. The simulation results demonstrate that the recognition accuracy is comparable to that of the traditional method after the compression process (94.62%), showing the potential of CTFC method for broadening the real-time application scenarios for event cameras.
Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognition

Huabin Liu,Weixian Lv,John See,Weiyao Lin

DOI: https://doi.org/10.48550/arXiv.2207.09759

2022-07-20

Computer Vision and Pattern Recognition

Abstract:A primary challenge faced in few-shot action recognition is inadequate video data for training. To address this issue, current methods in this field mainly focus on devising algorithms at the feature level while little attention is paid to processing input video data. Moreover, existing frame sampling strategies may omit critical action information in temporal and spatial dimensions, which further impacts video utilization efficiency. In this paper, we propose a novel video frame sampler for few-shot action recognition to address this issue, where task-specific spatial-temporal frame sampling is achieved via a temporal selector (TS) and a spatial amplifier (SA). Specifically, our sampler first scans the whole video at a small computational cost to obtain a global perception of video frames. The TS plays its role in selecting top-T frames that contribute most significantly and subsequently. The SA emphasizes the discriminative information of each frame by amplifying critical regions with the guidance of saliency maps. We further adopt task-adaptive learning to dynamically adjust the sampling strategy according to the episode task at hand. Both the implementations of TS and SA are differentiable for end-to-end optimization, facilitating seamless integration of our proposed sampler with most few-shot action recognition methods. Extensive experiments show a significant boost in the performances on various benchmarks including long-term videos.The code is available at https://github.com/R00Kie-Liu/Sampler
Learning Comprehensive Motion Representation for Action Recognition

Mingyu Wu,Boyuan Jiang,Donghao Luo,Junchi Yan,Yabiao Wang,Ying Tai,Chengjie Wang,Jilin Li,Feiyue Huang,Xiaokang Yang

DOI: https://doi.org/10.1609/aaai.v35i4.16400

2021-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:For action recognition learning, 2D CNN-based methods are efficient but may yield redundant features due to applying the same 2D convolution kernel to each frame. Recent efforts attempt to capture motion information by establishing interframe connections while still suffering the limited temporal receptive field or high latency. Moreover, the feature enhancement is often only performed by channel or space dimension in action recognition. To address these issues, we first devise a Channel-wise Motion Enhancement (CME) module to adaptively emphasize the channels related to dynamic information with a channel-wise gate vector. The channel gates generated by CME incorporate the information from all the other frames in the video. We further propose a Spatial-wise Motion Enhancement (SME) module to focus on the regions with the critical target in motion, according to the point-to-point similarity between adjacent feature maps. The intuition is that the change of background is typically slower than the motion area. Both CME and SME have clear physical meaning in capturing action clues. By integrating the two modules into the off-the-shelf 2D network, we finally obtain a Comprehensive Motion Representation (CMR) learning method for action recognition, which achieves competitive performance on Something-Something V1 & V2 and Kinetics-400. On the temporal reasoning datasets Something-Something V1 and V2, our method outperforms the current state-of-the-art by 2.3% and 1.9% when using 16 frames as input, respectively.
DMDC: a cross-attention network for dynamic mask-based dual-camera snapshot hyperspectral Photography

Zeyu Cai,Ziyu Zhang,Chengqian Jin,Feipeng Da

DOI: https://doi.org/10.1007/s00371-024-03700-z

IF: 2.835

2024-11-09

The Visual Computer

Abstract:Spectral images can enrich the material information of the reconstructed scene and have an essential role in computer visualization. Coded aperture snapshot spectral imaging (CASSI) for dynamic scenes still faces two problems in reconstruction: 1) The single input limits the network's performance. 2) The spatial light modulator's (SLM) performance has yet to fully develop due to the limitation of fixed mask coding. This paper proposes a cross-attention-based dual-stream network for a dual-camera CASSI system. We argue that RGB images and CASSI measurements are projections of spectral 3D cubes in different 2D spaces and that fusing the spatial features of RGB and the spectral features of CASSI improves the quality of the reconstruction. Upon that, we embed a dynamic mask module in front of the cross-attention-based dual-stream network to further improve the reconstruction quality of the system. Specifically, the dynamic mask module utilizes RGB images to pre-learn the spatial feature distribution of the scene. Then the dynamic mask module guides the SLM in encoding the CASSI. Finally, the RGB and CASSI images are reconstructed using a cross-attention-based dual-stream network to obtain high-quality reconstruction results. Comprehensive experiments on various datasets demonstrate the superior performance of our method. At similar speeds, our method provides a 4.0 dB improvement over existing SOTA methods on clean and noisy datasets. In the snapshot video imaging task, the single snapshot imaging time of DMDC-1stg is less than 50 ms, which verifies the feasibility of our method. (The code has been released at https://github.com/caizeyu1992/DMDC.)

computer science, software engineering
Dynamic Sampling Networks for Efficient Action Recognition in Videos.

Yin-Dong Zheng,Zhaoyang Liu,Tong Lu,Limin Wang

DOI: https://doi.org/10.1109/tip.2020.3007826

IF: 10.6

2020-01-01

IEEE Transactions on Image Processing

Abstract:The existing action recognition methods are mainly based on clip-level classifiers such as two-stream CNNs or 3D CNNs, which are trained from the randomly selected clips and applied to densely sampled clips during testing. However, this standard setting might be suboptimal for training classifiers and also requires huge computational overhead when deployed in practice. To address these issues, we propose a new framework for action recognition in videos, called Dynamic Sampling Networks (DSN), by designing a dynamic sampling module to improve the discriminative power of learned clip-level classifiers and as well increase the inference efficiency during testing. Specifically, DSN is composed of a sampling module and a classification module, whose objective is to learn a sampling policy to on-the-fly select which clips to keep and train a clip-level classifier to perform action recognition based on these selected clips, respectively. In particular, given an input video, we train an observation network in an associative reinforcement learning setting to maximize the rewards of the selected clips with a correct prediction. We perform extensive experiments to study different aspects of the DSN framework on four action recognition datasets: UCF101, HMDB51, THUMOS14, and ActivityNet v1.3. The experimental results demonstrate that DSN is able to greatly improve the inference efficiency by only using less than half of the clips, which can still obtain a slightly better or comparable recognition accuracy to the state-of-the-art approaches.

DCT-Cam for Single-shot Action Recognition

DoCam: Depth Sensing with an Optical Image Stabilization Supported RGB Camera.

Action Recognition Based on Discrete Cosine Transform by Optical Pixel-Wise Encoding

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Compressed Video Action Recognition with Dual-Stream and Dual-Modal Transformer

Action Recognition for Depth Video using Multi-view Dynamic Images

RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks

Full-resolution and full-dynamic-range coded aperture compressive temporal imaging

3D-TDC: A 3D temporal dilation convolution framework for video action recognition

Action Recognition and Benchmark Using Event Cameras.

Content-Aware Attention Network For Action Recognition

D3D: Dual 3-D Convolutional Network for Real-Time Action Recognition

Extreme Low-Resolution Action Recognition with Confident Spatial-Temporal Attention Transfer

Action Recognition with Multi-Scale Trajectory-Pooled 3D Convolutional Descriptors

DTCM: Joint Optimization of Dark Enhancement and Action Recognition in Videos

DA-CCD: A novel action representation by Deep Architecture of local depth feature

A Cascaded Timestamp-Free Event Camera Image Compression Method for Gesture Recognition

Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognition

Learning Comprehensive Motion Representation for Action Recognition

DMDC: a cross-attention network for dynamic mask-based dual-camera snapshot hyperspectral Photography

Dynamic Sampling Networks for Efficient Action Recognition in Videos.