Leveraging Expert Models for Training Deep Neural Networks in Scarce Data Domains: Application to Offline Handwritten Signature Verification

Dimitrios Tsourounis,Ilias Theodorakopoulos,Elias N. Zois,George Economou

2023-08-02

Abstract:This paper introduces a novel approach to leverage the knowledge of existing expert models for training new Convolutional Neural Networks, on domains where task-specific data are limited or unavailable. The presented scheme is applied in offline handwritten signature verification (OffSV) which, akin to other biometric applications, suffers from inherent data limitations due to regulatory restrictions. The proposed Student-Teacher (S-T) configuration utilizes feature-based knowledge distillation (FKD), combining graph-based similarity for local activations with global similarity measures to supervise student's training, using only handwritten text data. Remarkably, the models trained using this technique exhibit comparable, if not superior, performance to the teacher model across three popular signature datasets. More importantly, these results are attained without employing any signatures during the feature extraction training process. This study demonstrates the efficacy of leveraging existing expert models to overcome data scarcity challenges in OffSV and potentially other related domains.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of data scarcity in Offline Handwritten Signature Verification (OffSV). Specifically, it proposes a novel method that leverages the knowledge of existing expert models to train new Convolutional Neural Networks (CNNs), particularly in scenarios where task-specific data is limited or unavailable. The main contributions include:

1. **Application of Knowledge Distillation (KD)**: The paper introduces Data-Free Knowledge Distillation (DF-KD) to the OffSV field for the first time. By using Feature-based Knowledge Distillation (FKD), it combines geometric and global feature similarity to guide the learning process of the student model.

2. **Use of External Data**: Due to regulatory restrictions on signature datasets (such as GDPR), the paper opts to use handwritten text images as external data. These data have a similar structure to signatures and are easily accessible, thus avoiding potential issues with synthetic signature data.

3. **Integration of Local and Global Information**: The proposed method considers not only the spatial matching of intermediate layer activations but also incorporates a regularization loss function based on the global feature distribution. This enhances the similarity between features and reduces redundancy.

4. **Experimental Results**: The proposed approach performs excellently on three popular signature datasets, even surpassing the performance of the teacher model, and does so without using any real signature images during the entire training process.

In summary, this paper demonstrates how knowledge distillation can effectively overcome the problem of data scarcity in offline handwritten signature verification, providing valuable insights for the related field.