Subunit counting in membrane-bound proteins

Maximilian H. Ulbrich,E. Isacoff

DOI: https://doi.org/10.1038/nmeth1024

IF: 48

2007-04-01

Nature Methods

Abstract:

What problem does this paper attempt to address?

ASTM - an Attention Based Spatiotemporal Model for Video Prediction Using 3D Convolutional Neural Networks.

Zheng Chang,Xinfeng Zhang,Shanshe Wang,Siwei Ma,Yan Ye,Wen Gao

DOI: https://doi.org/10.1109/icme51207.2021.9428362

2021-01-01

Abstract:Video prediction has always been a challenging task in video representation learning due to the diversity of spatial-temporal evolution in videos. In this paper, we propose an Attention based SpatioTemporal Model for Video Prediction based on 3D Convolutional Neural Networks and Long Short-Term Memory (LSTM), which is named ASTM. In our method, we leverage both multi-term and short-term inter-frame dependencies in temporal domain to capture reliable motion information for videos. In particular, we design an Efficient Inter-Frame Attention Gate (EIFAG) to efficiently aggregate the multi-term inter-frame dependencies and integrate 3D convolutional operations into the proposed model to further improve the local perception to videos by capturing more accurate short-term temporal dependency. In addition, we make use of the multilayer Spatiotemporal LSTM (PredRNN) structure to preserve more spatial appearance details for videos. To evaluate the adaptability of our model on more complex real scenes, we collect a multi-level spatiotemporal (MLST) dataset. Experimental results show that the proposed model can achieve state-of-the-art performance on both widely used datasets and the proposed MLST dataset.
The prediction of head and eye movement for 360 degree images

Yucheng Zhu,Guangtao Zhai,Xiongkuo Min

DOI: https://doi.org/10.1016/j.image.2018.05.010

2018-11-01

Abstract:Estimating salient areas of visual stimuli which are liable to attract viewers’ visual attention is a challenging task because of the high complexity of cognitive behaviors in the brain. Many researchers have been dedicated to this field and obtained many achievements. Some application areas, ranging from computer vision, computer graphics, to multimedia processing, can benefit from saliency detection, considering that the detected saliency has depicted the visual importance of different areas of the visual stimuli. As for the 360 degree visual stimuli, images and videos should record the whole scene in the 3D world, so the resolutions of panoramic images and videos are usually very high. However, when watching 360 degree stimuli, observers can only see part of the scene in the view port, which is presented to the eyes of the observers through the Head Mounted Display (HMD). So sending the whole video, or rendering the whole scene may result in the waste of resources. Thus if we can predict the current field of view, then focuses can be put to the streaming and rendering of the scene in the current field of view. Further more, if we can predict salient areas in the scene, then more fine processing can be done to the visually important areas. The prediction of salient regions for traditional images and videos have been extensively studied. However, conventional saliency prediction methods are not fully adequate for 360 degree contents, because 360 degree stimuli own some unique characteristics. Related study in this area is limited. In this paper, we study the problem of predicting head movement, head–eye motion, and scanpath of viewers when they are watching 360 degree images in the commodity HMDs. Three types of data are specifically analyzed. The first is the head movement data, which can be regarded as the movement of the view port. The second is the head–eye motion data which combines the motion of the head and the movement of the eye within the view port. The third is the scan-paths data of observers in the entire panorama which record the position information as well as the time information. And our model is designed to predict the saliency maps for the first two, and the scanpaths for the last one. Experimental results demonstrate the effectiveness of our model.
Predicting 360° Video Saliency: A ConvLSTM Encoder-Decoder Network with Spatio-temporal Consistency

Zhaolin Wan,Han Qin,Ruiqin Xiong,Zhiyang Li,Xiaopeng Fan,Debin Zhao

DOI: https://doi.org/10.1109/jetcas.2024.3377096

IF: 5.877

2024-01-01

IEEE Journal on Emerging and Selected Topics in Circuits and Systems

Abstract:360° videos have been widely used with the development of virtual reality technology and triggered a demand to determine the most visually attractive objects in them, aka 360° video saliency prediction (VSP). While generative models, i.e., variational autoencoders or autoregressive models have proved their effectiveness in handling spatio-temporal data, utilizing them in 360° VSP is still challenging due to the problem of severe distortion and feature alignment inconsistency. In this study, we propose a novel spatio-temporal consistency generative network for 360° VSP. A dual-stream encoder-decoder architecture is adopted to process the forward and backward frame sequences of 360° videos simultaneously. Moreover, a deep autoregressive module termed as axial-attention based spherical ConvLSTM is designed in the encoder to memorize features with global-range spatial and temporal dependencies. Finally, motivated by the bias phenomenon in human viewing behavior, a temporal-convolutional Gaussian prior module is introduced to further improve the accuracy of the saliency prediction. Extensive experiments are conducted to evaluate our model and the state-of-the-art competitors, demonstrating that our model has achieved the best performance on the databases of PVS-HM and VR-Eyetracking.

engineering, electrical & electronic
Short-Term Trajectory Prediction for Full-Immersive Multiuser Virtual Reality with Redirected Walking

Filip Lemic,Jakob Struye,Jeroen Famaey

DOI: https://doi.org/10.48550/arXiv.2207.07520

2022-07-15

Networking and Internet Architecture

Abstract:Full-immersive multiuser Virtual Reality (VR) envisions supporting unconstrained mobility of the users in the virtual worlds, while at the same time constraining their physical movements inside VR setups through redirected walking. For enabling delivery of high data rate video content in real-time, the supporting wireless networks will leverage highly directional communication links that will "track" the users for maintaining the Line-of-Sight (LoS) connectivity. Recurrent Neural Networks (RNNs) and in particular Long Short-Term Memory (LSTM) networks have historically presented themselves as a suitable candidate for near-term movement trajectory prediction for natural human mobility, and have also recently been shown as applicable in predicting VR users' mobility under the constraints of redirected walking. In this work, we extend these initial findings by showing that Gated Recurrent Unit (GRU) networks, another candidate from the RNN family, generally outperform the traditionally utilized LSTMs. Second, we show that context from a virtual world can enhance the accuracy of the prediction if used as an additional input feature in comparison to the more traditional utilization of solely the historical physical movements of the VR users. Finally, we show that the prediction system trained on a static number of coexisting VR users be scaled to a multi-user system without significant accuracy degradation.
A Spherical Convolution Approach for Learning Long Term Viewport Prediction in 360 Immersive Video

Chenglei Wu,Rui-Xiao Zhang,Zhi Wang,Lifeng Sun

DOI: https://doi.org/10.1609/aaai.v34i01.7377

2020-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Viewport prediction for 360 video forecasts a viewer’s viewport when he/she watches a 360 video with a head-mounted display, which benefits many VR/AR applications such as 360 video streaming and mobile cloud VR. Existing studies based on planar convolutional neural network (CNN) suffer from the image distortion and split caused by the sphere-to-plane projection. In this paper, we start by proposing a spherical convolution based feature extraction network to distill spatial-temporal 360 information. We provide a solution for training such a network without a dedicated 360 image or video classification dataset. We differ with previous methods, which base their predictions on image pixel-level information, and propose a semantic content and preference based viewport prediction scheme. In this paper, we adopt a recurrent neural network (RNN) network to extract a user’s personal preference of 360 video content from minutes of embedded viewing histories. We utilize this semantic preference as spatial attention to help network find the “interested” regions on a future video. We further design a tailored mixture density network (MDN) based viewport prediction scheme, including viewport modeling, tailored loss function, etc, to improve efficiency and accuracy. Our extensive experiments demonstrate the rationality and performance of our method, which outperforms state-of-theart methods, especially in long-term prediction.
Optimizing Fixation Prediction Using Recurrent Neural Networks for 360$^{\circ }$ Video Streaming in Head-Mounted Virtual Reality

Ching-Ling Fan,Shou-Cheng Yen,Chun-Ying Huang,Cheng-Hsin Hsu

DOI: https://doi.org/10.1109/tmm.2019.2931807

IF: 7.3

2020-01-01

IEEE Transactions on Multimedia

Abstract:We study the problem of predicting the viewing probability of different parts of 360. videos when streaming them to head-mounted displays. We propose a fixation prediction network based on recurrent neural network, which leverages sensor and content features. The content features are derived by computer vision (CV) algorithms, which may suffer from inferior performance due to various types of distortion caused by diverse 360. video projection models. We propose a unified approach with overlapping virtual viewports to eliminate such negative effects, andwe evaluate our proposed solution using severalCValgorithms, such as saliency detection, face detection, and object detection. We find that overlapping virtual viewports increase the performance of these existing CV algorithms that were not trained for 360. videos. We next fine-tune our fixation prediction network with diverse design options, including: 1) with or without overlapping virtual viewports, 2) with or without future content features, and 3) different feature sampling rates. We empirically choose the best fixation prediction network and use it in a 360. video streaming system. We conduct extensive trace-driven simulations with a large-scale dataset to quantify the performance of the 360. video streaming system with different fixation prediction algorithms. The results show that our proposed fixation prediction network outperforms other algorithms in several aspects, such as: 1) achieving comparable video quality (average gaps between -0.05 and 0.92 dB), 2) consuming much less bandwidth (average bandwidth reduction by up to 8Mb/s), 3) reducing the rebuffering time (on average 40 s in bandwidth-limited 4G cellular networks), and 4) running in real-time (at most 124 ms).
Very Long Term Field of View Prediction for 360-degree Video Streaming

Chenge Li,Weixi Zhang,Yong Liu,Yao Wang

DOI: https://doi.org/10.48550/arXiv.1902.01439

2019-02-05

Abstract:360-degree videos have gained increasing popularity in recent years with the developments and advances in Virtual Reality (VR) and Augmented Reality (AR) technologies. In such applications, a user only watches a video scene within a field of view (FoV) centered in a certain direction. Predicting the future FoV in a long time horizon (more than seconds ahead) can help save bandwidth resources in on-demand video streaming while minimizing video freezing in networks with significant bandwidth variations. In this work, we treat the FoV prediction as a sequence learning problem, and propose to predict the target user's future FoV not only based on the user's own past FoV center trajectory but also other users' future FoV locations. We propose multiple prediction models based on two different FoV representations: one using FoV center trajectories and another using equirectangular heatmaps that represent the FoV center distributions. Extensive evaluations with two public datasets demonstrate that the proposed models can significantly outperform benchmark models, and other users' FoVs are very helpful for improving long-term predictions.

Computer Vision and Pattern Recognition
A Hybrid Transformer-LSTM Model With 3D Separable Convolution for Video Prediction

Mareeta Mathai,Ying Liu,Nam Ling

DOI: https://doi.org/10.1109/access.2024.3375365

IF: 3.9

2024-03-22

IEEE Access

Abstract:Video prediction is an essential vision task due to its wide applications in real-world scenarios. However, it is indeed challenging due to the inherent uncertainty and complex spatiotemporal dynamics of video content. Several state-of-the-art deep learning methods have achieved superior video prediction accuracy at the expense of huge computational cost. Hence, they are not suitable for devices with limitations in memory and computational resource. In the light of Green Artificial Intelligence (AI), more environment friendly deep learning solutions are desired to tackle the problem of large models and computational cost. In this work, we propose a novel video prediction network 3DTransLSTM, which adopts a hybrid transformer-long short-term memory (LSTM) structure to inherit the merits of both self-attention and recurrence. Three-dimensional (3D) depthwise separable convolutions are used in this hybrid structure to extract spatiotemporal features, meanwhile enhancing model efficiency. We conducted experimental studies on four popular video prediction datasets. Compared to existing methods, our proposed 3DTransLSTM achieved competitive frame prediction accuracy with significantly reduced model size, trainable parameters, and computational complexity. Moreover, we demonstrate the generalization ability of the proposed model by testing the model on dataset completely unseen in the training data.

computer science, information systems,telecommunications,engineering, electrical & electronic
Long short-term memory prediction of user's locomotion in virtual reality

Jesus Mayor,Pablo Calleja,Felix Fuentes-Hurtado

DOI: https://doi.org/10.1007/s10055-024-00962-9

IF: 4.697

2024-03-06

Virtual Reality

Abstract:Nowadays, there is still a challenge in virtual reality to obtain an accurate displacement prediction of the user. This could be a future key element to apply in the so-called redirected walking methods. Meanwhile, deep learning provides us with new tools to reach greater achievements in this type of prediction. Specifically, long short-term memory recurrent neural networks obtained promising results recently. This gives us clues to continue researching in this line to predict virtual reality user's displacement. This manuscript focuses on the collection of positional data and a subsequent new way to train a deep learning model to obtain more accurate predictions. The data were collected with 44 participants and it has been analyzed with different existing prediction algorithms. The best results were obtained with a new idea, the use of rotation quaternions and the three dimensions to train the previously existing models. The authors strongly believe that there is still much room for improvement in this research area by means of the usage of new deep learning models.

computer science, interdisciplinary applications, software engineering,imaging science & photographic technology
Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Yuhang Song,Mai Xu,Jianyi Wang,Minglang Qiao,Liangyu Huo,Zulin Wang

DOI: https://doi.org/10.1109/TPAMI.2018.2858783

2019-11-28

Abstract:Panoramic video provides immersive and interactive experience by enabling humans to control the field of view (FoV) through head movement (HM). Thus, HM plays a key role in modeling human attention on panoramic video. This paper establishes a database collecting subjects' HM in panoramic video sequences. From this database, we find that the HM data are highly consistent across subjects. Furthermore, we find that deep reinforcement learning (DRL) can be applied to predict HM positions, via maximizing the reward of imitating human HM scanpaths through the agent's actions. Based on our findings, we propose a DRL-based HM prediction (DHP) approach with offline and online versions, called offline-DHP and online-DHP. In offline-DHP, multiple DRL workflows are run to determine potential HM positions at each panoramic frame. Then, a heat map of the potential HM positions, named the HM map, is generated as the output of offline-DHP. In online-DHP, the next HM position of one subject is estimated given the currently observed HM position, which is achieved by developing a DRL algorithm upon the learned offline-DHP model. Finally, the experiments validate that our approach is effective in both offline and online prediction of HM positions for panoramic video, and that the learned offline-DHP model can improve the performance of online-DHP.

Computer Vision and Pattern Recognition,Machine Learning
Multi-level Motion Attention for Human Motion Prediction

Wei Mao,Miaomiao Liu,Mathieu Salzmann,Hongdong Li

DOI: https://doi.org/10.48550/arXiv.2106.09300

2021-06-17

Abstract:Human motion prediction aims to forecast future human poses given a historical motion. Whether based on recurrent or feed-forward neural networks, existing learning based methods fail to model the observation that human motion tends to repeat itself, even for complex sports actions and cooking activities. Here, we introduce an attention based feed-forward network that explicitly leverages this observation. In particular, instead of modeling frame-wise attention via pose similarity, we propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences. In this context, we study the use of different types of attention, computed at joint, body part, and full pose levels. Aggregating the relevant past motions and processing the result with a graph convolutional network allows us to effectively exploit motion patterns from the long-term history to predict the future poses. Our experiments on Human3.6M, AMASS and 3DPW validate the benefits of our approach for both periodical and non-periodical actions. Thanks to our attention model, it yields state-of-the-art results on all three datasets. Our code is available at <a class="link-external link-https" href="https://github.com/wei-mao-2019/HisRepItself" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Spatial and Temporal Visual Attention Prediction in Videos Using Eye Movement Data

Junwei Han,Liye Sun,Xintao Hu,Jungong Han,Ling Shao

DOI: https://doi.org/10.1016/j.neucom.2014.05.049

IF: 6

2014-01-01

Neurocomputing

Abstract:Visual attention detection in static images has achieved outstanding progress in recent years whereas much less effort has been devoted to learning visual attention in video sequences. In this paper, we propose a novel method to model spatial and temporal visual attention for videos respectively through learning from human gaze data. The spatial visual attention mainly predicts where viewers look in each video frame while the temporal visual attention measures which video frame is more likely to attract viewers׳ interest. Our underlying premise is that objects as well as their movements, instead of conventional contrast-related information, are major factors in dynamic scenes to drive visual attention. Firstly, the proposed models extract two types of bottom-up features derived from multi-scale object filter responses and spatiotemporal motion energy, respectively. Then, spatiotemporal gaze density and inter-observer gaze congruency are generated using a large collection of human-eye gaze data to form two training sets. Finally, prediction models of temporal visual attention and spatial visual attention are learned based on those two training sets and bottom-up features, respectively. Extensive evaluations on publicly available video benchmarks and applications in interestingness prediction of movie trailers demonstrate the effectiveness of the proposed work.
Fixation Prediction for 360 ° Video Streaming to Head-Mounted Displays

Ching-Ling Fan,Jean Lee,Wen-Chih Lo,Chun-Ying Huang,Kuan-Ta Chen,Cheng-Hsin Hsu

2017-01-01

Abstract:We study the problem of predicting the Field-of-Views (FoVs) of viewerswatching 360° videos using commodityHead-MountedDisplays (HMDs). Existing solutions either use the viewer’s current orientation to approximate the viewed tiles in the future, or extrapolate the FoVs using the historical orientations and dead-reckoning algorithms. In this paper, we develop xation prediction networks that concurrently leverage sensorand content-related features to predict the viewer xation in the future, which is quite dierent from the solutions in the literature. e sensor-related features include HMD orientations, while the content-related features include image saliencymaps andmotionmaps. We build a 360° video streaming testbed to HMDs, and recruit twenty-ve viewers to watch ten 360° videos. We then train and validate two design alternatives of our proposed networks, which allows us to identify the beer-performing design with the optimal parameter seings. Trace-driven simulation results show the merits of our proposed xation prediction networks compared to the existing solutions, including: (i) lower consumed bandwidth, (ii) shorter initial buering time, and (iii) short running time.
Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction

Zhenguang Liu,Shuang Wu,Shuyuan Jin,Qi Liu,Shouling Ji,Shijian Lu,Li Cheng

DOI: https://doi.org/10.1109/tpami.2021.3139918

IF: 23.6

2022-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Predicting human motion from historical pose sequence is crucial for a machine to succeed in intelligent interactions with humans. One aspect that has been obviated so far, is the fact that how we represent the skeletal pose has a critical impact on the prediction results. Yet there is no effort that investigates across different pose representation schemes. We conduct an indepth study on various pose representations with a focus on their effects on the motion prediction task. Moreover, recent approaches build upon off-the-shelf RNN units for motion prediction. These approaches process input pose sequence sequentially and inherently have difficulties in capturing long-term dependencies. In this paper, we propose a novel RNN architecture termed AHMR (Attentive Hierarchical Motion Recurrent network) for motion prediction which simultaneously models local motion contexts and a global context. We further explore a geodesic loss and a forward kinematics loss for the motion prediction task, which have more geometric significance than the widely employed L2 loss. Interestingly, we applied our method to a range of articulate objects including human, fish, and mouse. Empirical results show that our approach outperforms the state-of-the-art methods in short-term prediction and achieves much enhanced long-term prediction proficiency, such as retaining natural human-like motions over 50 seconds predictions. Our codes are released.

computer science, artificial intelligence,engineering, electrical & electronic
History Repeats Itself: Human Motion Prediction via Motion Attention

Wei Mao,Miaomiao Liu,Mathieu Salzmann

DOI: https://doi.org/10.48550/arXiv.2007.11755

2020-07-23

Abstract:Human motion prediction aims to forecast future human poses given a past motion. Whether based on recurrent or feed-forward neural networks, existing methods fail to model the observation that human motion tends to repeat itself, even for complex sports actions and cooking activities. Here, we introduce an attention-based feed-forward network that explicitly leverages this observation. In particular, instead of modeling frame-wise attention via pose similarity, we propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences. Aggregating the relevant past motions and processing the result with a graph convolutional network allows us to effectively exploit motion patterns from the long-term history to predict the future poses. Our experiments on Human3.6M, AMASS and 3DPW evidence the benefits of our approach for both periodical and non-periodical actions. Thanks to our attention model, it yields state-of-the-art results on all three datasets. Our code is available at <a class="link-external link-https" href="https://github.com/wei-mao-2019/HisRepItself" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning,Image and Video Processing
Improved AdaBoost for Virtual Reality Experience Prediction Based on Long Short-Term Memory Network

Wenhan Fan,Zhicheng Ding,Ruixin Huang,Chang Zhou,Xuyang Zhang

2024-05-17

Abstract:A classification prediction algorithm based on Long Short-Term Memory Network (LSTM) improved AdaBoost is used to predict virtual reality (VR) user experience. The dataset is randomly divided into training and test sets in the ratio of 7:3.During the training process, the model's loss value decreases from 0.65 to 0.31, which shows that the model gradually reduces the discrepancy between the prediction results and the actual labels, and improves the accuracy and generalisation ability.The final loss value of 0.31 indicates that the model fits the training data well, and is able to make predictions and classifications more accurately. The confusion matrix for the training set shows a total of 177 correct predictions and 52 incorrect predictions, with an accuracy of 77%, precision of 88%, recall of 77% and f1 score of 82%. The confusion matrix for the test set shows a total of 167 correct and 53 incorrect predictions with 75% accuracy, 87% precision, 57% recall and 69% f1 score. In summary, the classification prediction algorithm based on LSTM with improved AdaBoost shows good prediction ability for virtual reality user experience. This study is of great significance to enhance the application of virtual reality technology in user experience. By combining LSTM and AdaBoost algorithms, significant progress has been made in user experience prediction, which not only improves the accuracy and generalisation ability of the model, but also provides useful insights for related research in the field of virtual reality. This approach can help developers better understand user requirements, optimise virtual reality product design, and enhance user satisfaction, promoting the wide application of virtual reality technology in various fields.

Machine Learning
Human motion prediction with gated recurrent unit model of multi-dimensional input

Yue Yu,Niehao Tian,XiangYu Hao,Tao Ma,Chunguang Yang

DOI: https://doi.org/10.1007/s10489-021-02764-x

IF: 5.3

2021-09-16

Applied Intelligence

Abstract:The issue of human motion prediction aimed to predict sequences of joint positions or joint rotations of human skeleton has recently grown in importance. The Recurrent Neural Network is widely applied on the sequence prediction problems which has been proved effective. However it is difficult to train the model with human skeleton data of multi-dimensional as input, which would do naive forecasting to produce motionless sequence. To address the problem, it is a consensus that additional information will help to improve the accuracy, thus the angular velocities are extracted from the joint rotations as the input to enhance the prediction. Further more, this work adopts proper strategies on the basis of a stacked Gated Recurrent Unit network and verify them on the human motion prediction task. The experimental results show that our network outperforms the state-of-art on the short-term prediction task, and generates plausible action sequences in a relatively long period of time.

computer science, artificial intelligence
Optimizing Mobile-Friendly Viewport Prediction for Live 360-Degree Video Streaming

Lei Zhang,Tao Long,Weizhen Xu,Laizhong Cui,Jiangchuan Liu

2024-03-05

Abstract:Viewport prediction is the crucial task for adaptive 360-degree video streaming, as the bitrate control algorithms usually require the knowledge of the user's viewing portions of the frames. Various methods are studied and adopted for viewport prediction from less accurate statistic tools to highly calibrated deep neural networks. Conventionally, it is difficult to implement sophisticated deep learning methods on mobile devices, which have limited computation capability. In this work, we propose an advanced learning-based viewport prediction approach and carefully design it to introduce minimal transmission and computation overhead for mobile terminals. We also propose a model-agnostic meta-learning (MAML) based saliency prediction network trainer, which provides a few-sample fast training solution to obtain the prediction model by utilizing the information from the past models. We further discuss how to integrate this mobile-friendly viewport prediction (MFVP) approach into a typical 360-degree video live streaming system by formulating and solving the bitrate adaptation problem. Extensive experiment results show that our prediction approach can work in real-time for live video streaming and can achieve higher accuracies compared to other existing prediction methods on mobile end, which, together with our bitrate adaptation algorithm, significantly improves the streaming QoE from various aspects. We observe the accuracy of MFVP is 8.1$\%$ to 28.7$\%$ higher than other algorithms and achieves 3.73$\%$ to 14.96$\%$ higher average quality level and 49.6$\%$ to 74.97$\%$ less quality level change than other algorithms.

Multimedia,Image and Video Processing
Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning

Sangmin Lee,Hak Gu Kim,Dae Hwi Choi,Hyung-Il Kim,Yong Man Ro

DOI: https://doi.org/10.48550/arXiv.2104.00924

2021-04-02

Abstract:Our work addresses long-term motion context issues for predicting future frames. To predict the future precisely, it is required to capture which long-term motion context (e.g., walking or running) the input motion (e.g., leg movement) belongs to. The bottlenecks arising when dealing with the long-term motion context are: (i) how to predict the long-term motion context naturally matching input sequences with limited dynamics, (ii) how to predict the long-term motion context with high-dimensionality (e.g., complex motion). To address the issues, we propose novel motion context-aware video prediction. To solve the bottleneck (i), we introduce a long-term motion context memory (LMC-Memory) with memory alignment learning. The proposed memory alignment learning enables to store long-term motion contexts into the memory and to match them with sequences including limited dynamics. As a result, the long-term context can be recalled from the limited input sequence. In addition, to resolve the bottleneck (ii), we propose memory query decomposition to store local motion context (i.e., low-dimensional dynamics) and recall the suitable local context for each local part of the input individually. It enables to boost the alignment effects of the memory. Experimental results show that the proposed method outperforms other sophisticated RNN-based methods, especially in long-term condition. Further, we validate the effectiveness of the proposed network designs by conducting ablation studies and memory feature analysis. The source code of this work is available.

Computer Vision and Pattern Recognition
Attention-LSTM based prediction model for aircraft 4-D trajectory

Peiyan Jia,Huiping Chen,Lei Zhang,Daojun Han

DOI: https://doi.org/10.1038/s41598-022-19794-1

2022-09-15

Abstract:Aviation activities are constantly increasing as a result of the growth of the global economic system. How to increase airspace capacity within the limited airspace resources while ensuring smooth and safe aircraft operations is a challenge for civil aviation today. Air traffic safety is supported by accurate trajectory prediction. The way-points are relatively sparse, and there are many uncertain factors in the flight, which greatly increases the difficulty of trajectory prediction. So, it is vital to enhance trajectory prediction accuracy. An attention-LSTM trajectory prediction model is proposed in this paper, which is split into two parts. The time-series features of the flight trajectory are extracted in the initial stage using the long-short-term memory neural network (LSTM). In the second part, the attention mechanism is employed to process the extracted sequence features. The impact of secondary elements is reduced while the influence of primary ones is increased according to the attention mechanism. We used the advanced models in trajectory prediction as the comparison models, such as LSTM, support vector machine (SVM), back propagation (BP) neural network, Hidden Markov Model (HMM), and convolutional long-term memory neural network (CNN-LSTM). The model we proposed is superior to the model above based on quantitative analysis and comparison.

Subunit counting in membrane-bound proteins

ASTM - an Attention Based Spatiotemporal Model for Video Prediction Using 3D Convolutional Neural Networks.

The prediction of head and eye movement for 360 degree images

Predicting 360° Video Saliency: A ConvLSTM Encoder-Decoder Network with Spatio-temporal Consistency

Short-Term Trajectory Prediction for Full-Immersive Multiuser Virtual Reality with Redirected Walking

A Spherical Convolution Approach for Learning Long Term Viewport Prediction in 360 Immersive Video

Optimizing Fixation Prediction Using Recurrent Neural Networks for 360$^{\circ }$ Video Streaming in Head-Mounted Virtual Reality

Very Long Term Field of View Prediction for 360-degree Video Streaming

A Hybrid Transformer-LSTM Model With 3D Separable Convolution for Video Prediction

Long short-term memory prediction of user's locomotion in virtual reality

Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Multi-level Motion Attention for Human Motion Prediction

Spatial and Temporal Visual Attention Prediction in Videos Using Eye Movement Data

Fixation Prediction for 360 ° Video Streaming to Head-Mounted Displays

Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction

History Repeats Itself: Human Motion Prediction via Motion Attention

Improved AdaBoost for Virtual Reality Experience Prediction Based on Long Short-Term Memory Network

Human motion prediction with gated recurrent unit model of multi-dimensional input

Optimizing Mobile-Friendly Viewport Prediction for Live 360-Degree Video Streaming

Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning

Attention-LSTM based prediction model for aircraft 4-D trajectory