Exemplar-based video colorization with long-term spatiotemporal dependency
Siqi Chen,Xueming Li,Xianlin Zhang,Mingdao Wang,Yu Zhang,Jiatong Han,Yue Zhang
DOI: https://doi.org/10.1016/j.knosys.2023.111240
IF: 8.139
2023-11-29
Knowledge-Based Systems
Abstract:Exemplar-based video colorization is an essential technique for applications like old movie restoration. Although recent methods perform well in still scenes or scenes with regular movement, they always lack robustness in moving scenes due to their weak ability to model long-term dependency both spatially and temporally, leading to color fading, color discontinuity, or other artifacts. To solve this problem, we propose an exemplar-based video colorization framework with long-term spatiotemporal dependency. To enhance the long-term spatial dependency, a parallelized CNN-Transformer block and a double-head non-local operation are designed. The proposed CNN-Transformer block can better incorporate the long-term spatial dependency with local texture and structural features, and the double-head non-local operation further exploits the performance of the augmented feature. While for the long-term temporal dependency enhancement, we further introduce the novel Linkage subnet. The Linkage subnet propagates motion information across adjacent frame blocks and helps to maintain temporal continuity. Experiments demonstrate that our model outperforms recent state-of-the-art methods both quantitatively and qualitatively. Also, our model can generate more colorful, realistic and stabilized results, especially for scenes where objects change greatly and irregularly.
computer science, artificial intelligence