Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-body

Zeqing Wang,Qingyang Ma,Wentao Wan,Haojie Li,Keze Wang,Yonghong Tian
2024-11-21
Abstract:Recent improvements in visual synthesis have significantly enhanced the depiction of generated human photos, which are pivotal due to their wide applicability and demand. Nonetheless, the existing text-to-image or text-to-video models often generate low-quality human photos that might differ considerably from real-world body structures, referred to as "abnormal human bodies". Such abnormalities, typically deemed unacceptable, pose considerable challenges in the detection and repair of them within human photos. These challenges require precise abnormality recognition capabilities, which entail pinpointing both the location and the abnormality type. Intuitively, Visual Language Models (VLMs) that have obtained remarkable performance on various visual tasks are quite suitable for this task. However, their performance on abnormality detection in human photos is quite poor. Hence, it is quite important to highlight this task for the research community. In this paper, we first introduce a simple yet challenging task, i.e., \textbf{F}ine-grained \textbf{H}uman-body \textbf{A}bnormality \textbf{D}etection \textbf{(FHAD)}, and construct two high-quality datasets for evaluation. Then, we propose a meticulous framework, named HumanCalibrator, which identifies and repairs abnormalities in human body structures while preserving the other content. Experiments indicate that our HumanCalibrator achieves high accuracy in abnormality detection and accomplishes an increase in visual comparisons while preserving the other visual content.
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to detect and correct abnormal human body structures in generated human body images. Specifically, existing text - to - image or text - to - video models often produce situations that do not conform to the real - world human body structure when generating human body photos, which are called "abnormal human bodies". These abnormalities are generally considered unacceptable because they have significant differences from the real - world human body structure, which brings challenges to detecting and repairing these abnormalities. Therefore, this paper proposes a fine - grained human - body - anomaly - detection (FHAD) task and constructs two high - quality datasets for evaluation. In addition, the author proposes a framework named HumanCalibrator, which can identify and repair abnormalities in the human body structure while keeping other content unchanged. Experimental results show that HumanCalibrator has high accuracy in anomaly detection and has an improvement in visual comparison while maintaining the integrity of other visual content.