Siding Zeng,Jiangyan Yi,Jianhua Tao,Yujie Chen,Shan Liang,Yong Ren,Xiaohui Zhang
Abstract:When the task of locating manipulation regions in partially-fake audio (PFA) involves cross-domain datasets, the performance of deep learning models drops significantly due to the shift between the source and target domains. To address this issue, existing approaches often employ data augmentation before training. However, they overlook the characteristics in target domain that are absent in source domain. Inspired by the mixture-of-experts model, we propose an unsupervised method named Samples mining with Diversity and Entropy (SDE). Our method first learns from a collection of diverse experts that achieve great performance from different perspectives in the source domain, but with ambiguity on target samples. We leverage these diverse experts to select the most informative samples by calculating their entropy. Furthermore, we introduced a label generation method tailored for these selected samples that are incorporated in the training process in source domain integrating the target domain information. We applied our method to a cross-domain partially fake audio detection dataset, ADD2023Track2. By introducing 10% of unknown samples from the target domain, we achieved an F1 score of 43.84%, which represents a relative increase of 77.2% compared to the second-best method.
What problem does this paper attempt to address?
This paper attempts to solve the problem of locating the manipulated regions in partially forged audio (PFA) in cross - domain datasets. Specifically, when a deep - learning model trained from the source domain is applied to the target domain, due to the distribution differences between the source domain and the target domain, the model performance drops significantly. Existing methods usually adopt data augmentation techniques before training, but these methods ignore the features that exist in the target domain but are absent in the source domain.
To solve this problem, the authors propose an unsupervised domain adaptation method, called Samples mining with Diversity and Entropy (SDE). The main steps of the SDE method are as follows:
1. **Learning diverse expert models**: Through reverse knowledge distillation, learn a set of diverse expert models from the source domain. These expert models can interpret the source - domain data from different perspectives and perform well on the source domain, but are ambiguous on the target domain.
2. **Selecting the most informative samples**: Use these diverse expert models to calculate the information entropy of each sample, thereby selecting the most informative samples in the target domain. These samples contain features that do not exist in the source domain but exist in the target domain.
3. **Generating labels and introducing into the training process**: Generate soft labels for the selected most informative samples and incorporate these samples into the training process to combine the information of the target domain.
Through this method, the authors conducted experiments on the ADD2023Track2 dataset and achieved an F1 score of 43.84% by introducing only 10% of the target - domain samples, which is a 77.2% relative improvement compared to the second - best method.
### Formula summary
- **Cross - entropy loss function**:
\[
L = -\frac{1}{N} \sum_{j = 1}^{N} \sum_{i = 1}^{M} \left[ y(j,i) \log(p(j,i))+(1 - y(j,i)) \log(1 - p(j,i)) \right]
\]
where \( N \) is the total number of samples, \( M \) is the number of frames in each sample, \( y(j,i) \) is the true label of the \( i \)-th frame in the \( j \)-th sample, and \( p(j,i) \) is the probability that the model predicts the \( i \)-th frame to belong to the positive class.
- **Reverse reverse distillation loss**:
\[
L_{\text{Dis}}=\frac{1}{n} \sum_{w = 0}^{n} y_0\cdot\frac{1}{2}\max(0,\cos(h_{\text{now}},h_{\text{pre}}^w)-u)^2
\]
where \( h_{\text{now}} \) is the latent vector in the current training process, \( h_{\text{pre}}^w \) is the latent vector of the previous expert model, \( y_0 \) indicates whether they are different, \(\cos(h_{\text{now}},h_{\text{pre}}^w) \) is the cosine similarity, and \( u \) is the threshold.
- **Information entropy**:
\[
I_j=\sum_{i = 1}^{M} \left[ -P_i \log_2(P_i)-(1 - P_i) \log_2(1 - P_i) \right]
\]
where \( P_i \) is the probability that \( n \) experts predict the \( i \)-th frame to be manipulated, and the calculation formula is:
\[
P_i=\frac{m}{n}
\]
where \( m \) is the number of experts who predict to be manipulated, and \( n \) is the total number of experts.
Through the above methods, the authors successfully improve the detection performance of the manipulated regions in cross - domain partially forged audio.