Abstract:Zero-shot customized video generation has gained significant attention due to its substantial application potential. Existing methods rely on additional models to extract and inject reference subject features, assuming that the Video Diffusion Model (VDM) alone is insufficient for zero-shot customized video generation. However, these methods often struggle to maintain consistent subject appearance due to suboptimal feature extraction and injection techniques. In this paper, we reveal that VDM inherently possesses the force to extract and inject subject features. Departing from previous heuristic approaches, we introduce a novel framework that leverages VDM's inherent force to enable high-quality zero-shot customized video generation. Specifically, for feature extraction, we directly input reference images into VDM and use its intrinsic feature extraction process, which not only provides fine-grained features but also significantly aligns with VDM's pre-trained knowledge. For feature injection, we devise an innovative bidirectional interaction between subject features and generated content through spatial self-attention within VDM, ensuring that VDM has better subject fidelity while maintaining the diversity of the generated video. Experiments on both customized human and object video generation validate the effectiveness of our framework.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are the two core challenges in **zero - shot customized video generation**: **subject feature extraction** and **feature injection**. Specifically, existing methods rely on additional models to extract and inject the features of reference objects, which leads to several problems: 1. **Coarse feature extraction**: Existing methods are often only able to extract coarse - grained semantic features and cannot capture the details of the subject. 2. **Poor feature injection effect**: Due to the sub - optimal feature extraction and injection techniques, it is difficult to ensure the appearance consistency of the subject in the generated video. 3. **High training cost**: Additional training parameters are required, increasing the computational cost and complexity. To solve these problems, the author proposes a new framework - **VideoMaker**, which utilizes the inherent capabilities of the video diffusion model (VDM) to achieve high - quality zero - shot customized video generation. Specific improvements include: - **Feature extraction**: Directly input the reference image into the VDM, utilize its internal feature extraction process to provide fine - grained features, and be highly consistent with the pre - trained knowledge of the VDM. - **Feature injection**: Through the spatial self - attention mechanism in the VDM, perform two - way interaction with the subject features when generating content, ensuring the fidelity of the subject while maintaining the diversity of the generated video. Through this method, VideoMaker not only avoids additional training modules, but also can extract and inject subject features more effectively, thus achieving high - quality zero - shot customized video generation.

VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models