Abstract:Recent advancements in visual generation technologies have markedly increased the scale and availability of video datasets, which are crucial for training effective video generation models. However, a significant lack of high-quality, human-centric video datasets presents a challenge to progress in this field. To bridge this gap, we introduce \textbf{OpenHumanVid}, a large-scale and high-quality human-centric video dataset characterized by precise and detailed captions that encompass both human appearance and motion states, along with supplementary human motion conditions, including skeleton sequences and speech audio. To validate the efficacy of this dataset and the associated training strategies, we propose an extension of existing classical diffusion transformer architectures and conduct further pretraining of our models on the proposed dataset. Our findings yield two critical insights: First, the incorporation of a large-scale, high-quality dataset substantially enhances evaluation metrics for generated human videos while preserving performance in general video generation tasks. Second, the effective alignment of text with human appearance, human motion, and facial motion is essential for producing high-quality video outputs. Based on these insights and corresponding methodologies, the straightforward extended network trained on the proposed dataset demonstrates an obvious improvement in the generation of human-centric videos. The source code and the dataset are available at: \href{<a class="link-external link-https" href="https://fudan-generative-vision.github.io/OpenHumanVid" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://fudan-generative-vision.github.io/OpenHumanVid" rel="external noopener nofollow">this https URL</a>}.

FCVID : Fudan-Columbia Video Dataset

LSVC2017

Unsupervised Teacher-Student Model for Large-Scale Video Retrieval.

Consumer Video Understanding

Proceedings of the Workshop on Large-Scale Video Classification Challenge

FSVVD: A Dataset of Full Scene Volumetric Video

VAD: A Video Affective Dataset with Danmu

FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos

Towards Open-Vocabulary Video Instance Segmentation

An Open Dataset for Video Coding for Machines Standardization.

Short Film Dataset (SFD): A Benchmark for Story-Level Video Understanding

Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis

A Content-Driven Micro-Video Recommendation Dataset at Scale

RWF-2000: An Open Large Scale Video Database for Violence Detection

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

VVA: Video Values Analysis.

OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation

A Multimodal Sentiment Dataset for Video Recommendation

Fudan University at TRECVID 2010 : Semantic Indexing.

FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platforms

CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios