EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation

Xiaofeng Wang,Kang Zhao,Feng Liu,Jiayu Wang,Guosheng Zhao,Xiaoyi Bao,Zheng Zhu,Yingya Zhang,Xingang Wang

2024-11-13

Abstract:Video generation has emerged as a promising tool for world simulation, leveraging visual data to replicate real-world environments. Within this context, egocentric video generation, which centers on the human perspective, holds significant potential for enhancing applications in virtual reality, augmented reality, and gaming. However, the generation of egocentric videos presents substantial challenges due to the dynamic nature of egocentric viewpoints, the intricate diversity of actions, and the complex variety of scenes encountered. Existing datasets are inadequate for addressing these challenges effectively. To bridge this gap, we present EgoVid-5M, the first high-quality dataset specifically curated for egocentric video generation. EgoVid-5M encompasses 5 million egocentric video clips and is enriched with detailed action annotations, including fine-grained kinematic control and high-level textual descriptions. To ensure the integrity and usability of the dataset, we implement a sophisticated data cleaning pipeline designed to maintain frame consistency, action coherence, and motion smoothness under egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable of generating egocentric videos driven simultaneously by action descriptions and kinematic control signals. The EgoVid-5M dataset, associated action annotations, and all data cleansing metadata will be released for the advancement of research in egocentric video generation.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the challenges faced in generating egocentric (first-person perspective) videos. Specifically, these challenges include:

1. **Dynamic Perspective**: First-person perspective videos are difficult to generate due to the constantly changing viewpoint.
2. **Action Diversity**: The actions included in first-person perspective videos are very complex and diverse.
3. **Scene Diversity**: The scenes encountered in first-person perspective videos are also very rich, ranging from home environments to outdoor activities, and office activities.

Existing datasets are insufficient in these aspects and cannot effectively address the above challenges. Therefore, this paper proposes the **EgoVid-5M** dataset, a high-quality dataset meticulously curated for generating first-person perspective videos. EgoVid-5M contains 5 million first-person perspective video clips and provides detailed action annotations, including fine motion control and high-level text descriptions. Additionally, the dataset employs strict data cleaning strategies to ensure frame consistency, action coherence, and motion smoothness.

By introducing the EgoVid-5M dataset, the authors aim to advance research and applications in first-person perspective video generation, particularly in fields such as virtual reality (VR), augmented reality (AR), and gaming.