Abstract:Unmanned aerial vehicles (UAVs) are now widely applied to data acquisition due to its low cost and fast mobility. With the increasing volume of aerial videos, the demand for automatically parsing these videos is surging. To achieve this, current researches mainly focus on extracting a holistic feature with convolutions along both spatial and temporal dimensions. However, these methods are limited by small temporal receptive fields and cannot adequately capture long-term temporal dependencies which are important for describing complicated dynamics. In this paper, we propose a novel deep neural network, termed FuTH-Net, to model not only holistic features, but also temporal relations for aerial video classification. Furthermore, the holistic features are refined by the multi-scale temporal relations in a novel fusion module for yielding more discriminative video representations. More specially, FuTH-Net employs a two-pathway architecture: (1) a holistic representation pathway to learn a general feature of both frame appearances and shortterm temporal variations and (2) a temporal relation pathway to capture multi-scale temporal relations across arbitrary frames, providing long-term temporal dependencies. Afterwards, a novel fusion module is proposed to spatiotemporal integrate the two features learned from the two pathways. Our model is evaluated on two aerial video classification datasets, ERA and Drone-Action, and achieves the state-of-the-art results. This demonstrates its effectiveness and good generalization capacity across different recognition tasks (event classification and human action recognition). To facilitate further research, we release the code at <a class="link-external link-https" href="https://gitlab.lrz.de/ai4eo/reasoning/futh-net" rel="external noopener nofollow">this https URL</a>.

Fault Prediction of Unmanned Aerial Vehicles Based on Entropy Weight Fusion and Temporal Graph Convolutional Network with Non-Euclidian Structure

Graph Attention Network-Based Fault Detection for UAVs With Multivariant Time Series Flight Data

Advancing UAV Sensor Fault Diagnosis Based on Prior Knowledge and Graph Convolutional Network

A Deep Learning Approach for Fault-Tolerant Data Fusion Applied to UAV Position and Orientation Estimation

Unmanned Aerial Vehicle Fault Diagnosis Based on Ensemble Deep Learning Model

Data-driven multivariate regression-based anomaly detection and recovery of unmanned aerial vehicle flight data

An Intelligent Fault Diagnosis Approach for Multirotor UAVs Based on Deep Neural Network of Multi-Resolution Transform Features

A Novel Dual Attention Convolutional Neural Network Based on Multisensory Frequency Features for Unmanned Aerial Vehicle Rotor Fault Diagnosis

On-board Deep-learning-based Unmanned Aerial Vehicle Fault Cause Detection and Identification

Spatio-temporal correlation-based multiple regression for anomaly detection and recovery of unmanned aerial vehicle flight data

Unmanned Aerial Vehicles anomaly detection model based on sensor information fusion and hybrid multimodal neural network

Structural fault diagnosis of UAV based on convolutional neural network and data processing technology

FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification

Building on prior lightweight CNN model combined with LSTM-AM framework to guide fault detection in fixed-wing UAVs

CLGNN: UAV Fault Diagnosis Via Causal Learning and Graph Neural Network

Spatial Information-Aware Flight Safety Forecasting Model for Unmanned Aerial Vehicles Based on Deep Learning and Grey Analysis

Predicting the Health Status of an Unmanned Aerial Vehicles Data-Link System Based on a Bayesian Network

Anomaly Detection for Data from Unmanned Systems via Improved Graph Neural Networks with Attention Mechanism

A2DJP: A Two Graph-based Component Fused Learning Framework for Urban Anomaly Distribution and Duration Joint-Prediction

A Lightweight and Explainable Data-Driven Scheme for Fault Detection of Aerospace Sensors

Typical fault prediction method for wind turbines based on an improved stacked autoencoder network