Knowledge distillation of face recognition via attention cosine similarity review

Zhuo Wang,SuWen Zhao,WanYi Guo
DOI: https://doi.org/10.1049/cvi2.12288
IF: 1.484
2024-06-01
IET Computer Vision
Abstract:The authors propose a cross‐stage connection review of the attention cosine similarity knowledge distillation method. This method leverages the teacher's shallow multi‐stage attention maps to guide deep single‐stage attention map in the student model. In the AMFM module, multi‐stage attention map is fused, and in the HCCSL module, the spatial pyramid pooling technique is employed obtain multi‐level of attention map information, different levels of attention context information and attention map are calculated cosine similarity loss. The proposed algorithm superior performance is compared to sota methods. Deep learning‐based face recognition models have demonstrated remarkable performance in benchmark tests, and knowledge distillation technology has been frequently accustomed to obtain high‐precision real‐time face recognition models specifically designed for mobile and embedded devices. However, in recent years, the knowledge distillation methods for face recognition, which mainly focus on feature or logit knowledge distillation techniques, neglect the attention mechanism that play an important role in the domain of neural networks. An innovation cross‐stage connection review path of the attention cosine similarity knowledge distillation method that unites the attention mechanism with review knowledge distillation method is proposed. This method transfers the attention map obtained from the teacher network to the student through a cross‐stage connection path. The efficacy and excellence of the proposed algorithm are demonstrated in popular benchmark tests.
computer science, artificial intelligence,engineering, electrical & electronic
What problem does this paper attempt to address?