Progressive Sign Language Video Translation Model for Real-World Complex Background Environments

Jingchen Zou,Jianqiang Li,Xi Xu,Yuning Huang,Jing Tang,Changwei Song,Linna Zhao,Wenxiu Cheng,Chujie Zhu,Suqin Liu

DOI: https://doi.org/10.1109/compsac61105.2024.00076

2024-01-01

Abstract:Sign language video translation, which converts sign language information into textual expressions, play a vital role in breaking down the language communication barrier between deaf and healthy people. Existing translation methods are mainly focus on the single and pure background. However, the background in real-world environments is always complex, and these methods are difficult to achieve effective recognition results. To address this issue, we have exploratively constructed a real-world complex background sign language dataset (CBSL), containing sign language videos captured in various authentic environments (e.g., different backgrounds and lighting conditions). Based on this, we propose a progressive sign language translation model to effectively separate sign language users from the background and reduce environmental interference, thus significantly improving the generalization ability. Our proposed method significantly outperforms various comparative methods across all performance metrics on the CBSL dataset. Furthermore, on the publicly available Chinese Sign Language Continuous Recognition dataset(CSL), our method performs comparably to the current state-of-the-art (SOTA).