Use of peer role models to increase Pap smear and mammogram screening in Mexican-American and black women.

Lucina Suarez,Donna C Nichols,Cody A. Brady

DOI: https://doi.org/10.1016/S0749-3797(18)30705-0

IF: 6.604

1993-09-01

American Journal of Preventive Medicine

Abstract:

What problem does this paper attempt to address?

Learning Student Networks in the Wild

Hanting Chen,Tianyu Guo,Chang Xu,Wenshuo Li,Chunjing Xu,Chao Xu,Yunhe Wang

DOI: https://doi.org/10.1109/CVPR46437.2021.00636

2021-01-01

Abstract:Data free learning for student networks is a new paradigm for solving users' anxiety caused by the privacy problem of using original training data. Since the architectures of modern convolutional neural networks (CNNs) are compact and sophisticated, the alternative images or meta-data generated from the teacher network are often broken. Thus, the student network cannot achieve the comparable performance to that of the pre-trained teacher network especially on the large-scale image dataset. Different to previous works, we present to maximally utilize the massive available unlabeled data in the wild. Specifically, we first thoroughly analyze the output differences between teacher and student network on the original data and develop a data collection method. Then, a noisy knowledge distillation algorithm is proposed for achieving the performance of the student network. In practice, an adaptation matrix is learned with the student network for correcting the label noise produced by the teacher network on the collected unlabeled images. The effectiveness of our DFND (DataFree Noisy Distillation) method is then verified on several benchmarks to demonstrate its superiority over state-of-the-art data-free distillation methods. Experiments on various datasets demonstrate that the student networks learned by the proposed method can achieve comparable performance with those using the original dataset.
Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Xiaotian Yu,Shengxuming Zhang,Lingxiang Jia,Yuexuan Wang,Mingli Song,Zunlei Feng

DOI: https://doi.org/10.1016/j.neucom.2023.126829

IF: 6

2024-01-01

Neurocomputing

Abstract:Recently, many benchmark datasets have been found to contain noisy labels caused by unavoidable human mistakes. Many researchers propose new noise-aware loss functions to achieve robust classification performance. However, we discover that existing noise-aware loss functions cannot fully heal the damage caused by the noise. On the other hand, some methods filter out low confidence samples and train new models, whereas the filtered samples contain both noisy and hard samples that are critical for the robustness of models. Based on the above two discoveries, we devised the Noise-aware Network (NA-Net) for robust training with noisy labels. Each layer of NA-Net contains three groups of convolution kernels responsible for mix samples, clean samples, and noisy samples, termed as mix-kernels, clean-kernels, and noise-kernels, respectively. Mix-kernels are used for finding the clean samples with a newly devised noise-immune (NI) loss function; clean-kernels are targeted at learning better features without being misguided by noise; noise-kernels are trained by the remaining samples to rectify wrong labels for the next iteration. Meanwhile, for increasing the classification performance of mix-kernels, the extracted feature maps of clean-kernels without being poisoned are combined as the input of mix-kernels of the next layer. Also, the knowledge distillation strategy is adopted to distill the knowledge from clean-kernels to the noise-kernels. Extensive experiments demonstrate that the mutual promotion of three groups of kernels in NA-Net achieves state-of-the-art performance on both artificial noisy datasets and real-world datasets.
DCCD: Reducing Neural Network Redundancy Via Distillation

Yuang Liu,Jun Chen,Yong Liu

DOI: https://doi.org/10.1109/tnnls.2023.3238337

IF: 14.255

2024-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.
Remote Sensing Image Scene Classification with Noisy Label Distillation

Rui Zhang,Zhenghao Chen,Sanxing Zhang,Fei Song,Gang Zhang,Quancheng Zhou,Tao Lei

DOI: https://doi.org/10.3390/rs12152376

IF: 5

2020-07-24

Remote Sensing

Abstract:The widespread applications of remote sensing image scene classification-based Convolutional Neural Networks (CNNs) are severely affected by the lack of large-scale datasets with clean annotations. Data crawled from the Internet or other sources allows for the most rapid expansion of existing datasets at a low-cost. However, directly training on such an expanded dataset can lead to network overfitting to noisy labels. Traditional methods typically divide this noisy dataset into multiple parts. Each part fine-tunes the network separately to improve performance further. These approaches are inefficient and sometimes even hurt performance. To address these problems, this study proposes a novel noisy label distillation method (NLD) based on the end-to-end teacher-student framework. First, unlike general knowledge distillation methods, NLD does not require pre-training on clean or noisy data. Second, NLD effectively distills knowledge from labels across a full range of noise levels for better performance. In addition, NLD can benefit from a fully clean dataset as a model distillation method to improve the student classifier’s performance. NLD is evaluated on three remote sensing image datasets, including UC Merced Land-use, NWPU-RESISC45, AID, in which a variety of noise patterns and noise amounts are injected. Experimental results show that NLD outperforms widely used directly fine-tuning methods and remote sensing pseudo-labeling methods.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
Evaluation of muscular load of m. biceps brachii in isometric contraction with the use of microvibration.

K. Sakamoto,T. Usui

DOI: https://doi.org/10.2114/AHS1983.7.23

Abstract:
Learning from a Teacher using Unlabeled Data

Gaurav Menghani,Sujith Ravi

DOI: https://doi.org/10.48550/arXiv.1911.05275

2019-11-13

Abstract:Knowledge distillation is a widely used technique for model compression. We posit that the teacher model used in a distillation setup, captures relationships between classes, that extend beyond the original dataset. We empirically show that a teacher model can transfer this knowledge to a student model even on an {\it out-of-distribution} dataset. Using this approach, we show promising results on MNIST, CIFAR-10, and Caltech-256 datasets using unlabeled image data from different sources. Our results are encouraging and help shed further light from the perspective of understanding knowledge distillation and utilizing unlabeled data to improve model quality.

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition
Distribution Shift Matters for Knowledge Distillation with Webly Collected Images

Jialiang Tang,Shuo Chen,Gang Niu,Masashi Sugiyama,Chen Gong

2023-07-21

Abstract:Knowledge distillation aims to learn a lightweight student network from a pre-trained teacher network. In practice, existing knowledge distillation methods are usually infeasible when the original training data is unavailable due to some privacy issues and data management considerations. Therefore, data-free knowledge distillation approaches proposed to collect training instances from the Internet. However, most of them have ignored the common distribution shift between the instances from original training data and webly collected data, affecting the reliability of the trained student network. To solve this problem, we propose a novel method dubbed ``Knowledge Distillation between Different Distributions" (KD$^{3}$), which consists of three components. Specifically, we first dynamically select useful training instances from the webly collected data according to the combined predictions of teacher network and student network. Subsequently, we align both the weighted features and classifier parameters of the two networks for knowledge memorization. Meanwhile, we also build a new contrastive learning block called MixDistribution to generate perturbed data with a new distribution for instance alignment, so that the student network can further learn a distribution-invariant representation. Intensive experiments on various benchmark datasets demonstrate that our proposed KD$^{3}$ can outperform the state-of-the-art data-free knowledge distillation approaches.

Computer Vision and Pattern Recognition,Artificial Intelligence
Improve Knowledge Distillation via Label Revision and Data Selection

Weichao Lan,Yiu-ming Cheung,Qing Xu,Buhua Liu,Zhikai Hu,Mengke Li,Zhenghua Chen

2024-04-03

Abstract:Knowledge distillation (KD) has become a widely used technique in the field of model compression, which aims to transfer knowledge from a large teacher model to a lightweight student model for efficient network development. In addition to the supervision of ground truth, the vanilla KD method regards the predictions of the teacher as soft labels to supervise the training of the student model. Based on vanilla KD, various approaches have been developed to further improve the performance of the student model. However, few of these previous methods have considered the reliability of the supervision from teacher models. Supervision from erroneous predictions may mislead the training of the student model. This paper therefore proposes to tackle this problem from two aspects: Label Revision to rectify the incorrect supervision and Data Selection to select appropriate samples for distillation to reduce the impact of erroneous supervision. In the former, we propose to rectify the teacher's inaccurate predictions using the ground truth. In the latter, we introduce a data selection technique to choose suitable training samples to be supervised by the teacher, thereby reducing the impact of incorrect predictions to some extent. Experiment results demonstrate the effectiveness of our proposed method, and show that our method can be combined with other distillation approaches, improving their performance.

Machine Learning,Artificial Intelligence
GEIKD : Self-knowledge distillation based on gated ensemble networks and influences-based label noise removal

Fuchang Liu,Yu Wang,Zheng Li,Zhigeng Pan

DOI: https://doi.org/10.1016/j.cviu.2023.103771

IF: 4.886

2023-07-12

Computer Vision and Image Understanding

Abstract:Self-distillation has gained widespread attention in recent years because it progressively transfers the knowledge in end-to-end training schemes within one network. However, self-distillation methods are susceptible to label noise hence leading to poor generalization performance. To address this problem, this paper proposes a novel self-distillation method, called GEIKD , which combines a gated ensemble self-teacher network and the influences-based label noise removal. Specifically, we design a gated ensemble self-teacher network composed of multiple teacher branches, which allows a gated fused knowledge based on a weighted bi-directional feature pyramid network. Moreover, we introduce influences estimation into the distillation process to quantify the effect of noisy labels on the distillation loss, and then reject the unfavorable instances as noisy labeled samples according to the calculated influences. Our influences-based label noise removal can be integrated with any existing knowledge distillation training schemes. The impact of noisy labels on knowledge distillation can be significantly alleviated by the proposed noisy instances removal with little extra training efforts. Experiments show that the proposed GEIKD method outperforms the state-of-the-art methods on CIFAR-100, TinyimageNet and fine-grained datasets CUB200, MIT-67, Stanford40 and FERC dataset, using clean data and data with noisy labels.

computer science, artificial intelligence,engineering, electrical & electronic
Small Scale Data-Free Knowledge Distillation

He Liu,Yikai Wang,Huaping Liu,Fuchun Sun,Anbang Yao

2024-06-12

Abstract:Data-free knowledge distillation is able to utilize the knowledge learned by a large teacher network to augment the training of a smaller student network without accessing the original training data, avoiding privacy, security, and proprietary risks in real applications. In this line of research, existing methods typically follow an inversion-and-distillation paradigm in which a generative adversarial network on-the-fly trained with the guidance of the pre-trained teacher network is used to synthesize a large-scale sample set for knowledge distillation. In this paper, we reexamine this common data-free knowledge distillation paradigm, showing that there is considerable room to improve the overall training efficiency through a lens of ``small-scale inverted data for knowledge distillation". In light of three empirical observations indicating the importance of how to balance class distributions in terms of synthetic sample diversity and difficulty during both data inversion and distillation processes, we propose Small Scale Data-free Knowledge Distillation SSD-KD. In formulation, SSD-KD introduces a modulating function to balance synthetic samples and a priority sampling function to select proper samples, facilitated by a dynamic replay buffer and a reinforcement learning strategy. As a result, SSD-KD can perform distillation training conditioned on an extremely small scale of synthetic samples (e.g., 10X less than the original training data scale), making the overall training efficiency one or two orders of magnitude faster than many mainstream methods while retaining superior or competitive model performance, as demonstrated on popular image classification and semantic segmentation benchmarks. The code is available at <a class="link-external link-https" href="https://github.com/OSVAI/SSD-KD" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
Student as an Inherent Denoiser of Noisy Teacher

Jiachen Zhao

2023-12-16

Abstract:Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by teacher models are usually noisy and may influence KD performance. This study delves into KD with noisy teachers and uncovers that the student model can already generate more accurate predictions than the teacher labels used to train it during KD, indicating its inherent ability to denoise noisy teacher labels. Motivated by this finding, we propose Peer-Advised KD to improve vanilla KD from noisy teachers. Experiments show that Peer-Advised KD can outperform LLM by approximately 5% with 50 human-labeled data, and even competitive to standard supervised finetuning with 750 human-labeled data.

Machine Learning,Computation and Language
Collaborative Knowledge Distillation

Weiwei Zhang,Yufeng Guo,Junhuang Wang,Jianqing Zhu,Huanqiang Zeng

DOI: https://doi.org/10.1109/tcsvt.2024.3377251

IF: 5.859

2024-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Existing research on knowledge distillation has primarily concentrated on the task of facilitating student networks in acquiring the complete knowledge imparted by teacher networks. However, recent studies have shown that good networks are not suitable for acting as teachers, and there is a positive correlation between distillation performance and teacher prediction uncertainty. To address this finding, this paper thoroughly analyzes in depth the reasons why the teacher network affects the distillation performance, gives full play to the participation of the student network in the process of knowledge distillation, and assists the teacher network in distilling the knowledge that is suitable for their learning. In light of this premise, a novel approach known as Collaborative Knowledge Distillation (CKD) is introduced, which is founded upon the concept of "Tailoring the Teaching to the Individual". Compared with Baseline, this paper’s method improves students’ accuracy by an average of 3.42% in CIFAR-100 experiments, and by an average of 1.71% compared with the classical Knowledge Distillation (KD) method. The ImageNet experiments conducted revealed a significant improvement of 2.04% in the Top-1 accuracy of the students.

engineering, electrical & electronic
Teacher-student collaborative knowledge distillation for image classification

Chuanyun Xu,Wenjian Gao,Tian Li,Nanlan Bai,Gang Li,Yang Zhang

DOI: https://doi.org/10.1007/s10489-022-03486-4

IF: 5.3

2022-05-04

Applied Intelligence

Abstract:A single model usually cannot learn all the appropriate features with limited data, thus leading to poor performance when test data are used. To improve model performance, we propose a teacher-student collaborative knowledge distillation (TSKD) method based on knowledge distillation and self-distillation. The method consists of two parts: learning in the teacher network and self-teaching in the student network. Learning in the teacher network allows the student network to use knowledge from the teacher network. Self-teaching in the student network is to build a multi-exit network based on self-distillation and provide deep features as supervised information for training. In the inference stage, we use ensembles to vote on the classification results of multiple sub-models in the student network. The experimental results demonstrate the superior performance of our method compared with a traditional knowledge distillation method and a self-distillation-based multi-exit network.

computer science, artificial intelligence
Synthetic data generation method for data-free knowledge distillation in regression neural networks

Tianxun Zhou,Keng-Hwee Chiam

DOI: https://doi.org/10.1016/j.eswa.2023.120327

2023-05-10

Abstract:Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger neural network as much as possible. Existing methods of knowledge distillation are mostly applicable for classification tasks. Many of them also require access to the data used to train the teacher model. To address the problem of knowledge distillation for regression tasks under the absence of original training data, previous work has proposed a data-free knowledge distillation method where synthetic data are generated using a generator model trained adversarially against the student model. These synthetic data and their labels predicted by the teacher model are then used to train the student model. In this study, we investigate the behavior of various synthetic data generation methods and propose a new synthetic data generation strategy that directly optimizes for a large but bounded difference between the student and teacher model. Our results on benchmark and case study experiments demonstrate that the proposed strategy allows the student model to learn better and emulate the performance of the teacher model more closely.

Machine Learning
Fixing the Teacher-Student Knowledge Discrepancy in Distillation

Jiangfan Han,Mengya Gao,Yujie Wang,Quanquan Li,Hongsheng Li,Xiaogang Wang

DOI: https://doi.org/10.48550/arXiv.2103.16844

2021-03-31

Abstract:Training a small student network with the guidance of a larger teacher network is an effective way to promote the performance of the student. Despite the different types, the guided knowledge used to distill is always kept unchanged for different teacher and student pairs in previous knowledge distillation methods. However, we find that teacher and student models with different networks or trained from different initialization could have distinct feature representations among different channels. (e.g. the high activated channel for different categories). We name this incongruous representation of channels as teacher-student knowledge discrepancy in the distillation process. Ignoring the knowledge discrepancy problem of teacher and student models will make the learning of student from teacher more difficult. To solve this problem, in this paper, we propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student and provides the best suitable knowledge to different student networks for distillation. Extensive experiments on different datasets (CIFAR100, ImageNet, COCO) and tasks (image classification, object detection) reveal the widely existing knowledge discrepancy problem between teachers and students and demonstrate the effectiveness of our proposed method. Our method is very flexible that can be easily combined with other state-of-the-art approaches.

Computer Vision and Pattern Recognition
Knowledge Distillation Meets Self-Supervision

Guodong Xu,Ziwei Liu,Xiaoxiao Li,Chen Change Loy

DOI: https://doi.org/10.48550/arXiv.2006.07114

2020-07-13

Abstract:Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an important technique for model compression and transfer learning. Unlike previous works that exploit architecture-specific cues such as activation and attention for distillation, here we wish to explore a more general and model-agnostic approach for extracting "richer dark knowledge" from the pre-trained teacher model. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. For example, when performing contrastive learning between transformed entities, the noisy predictions of the teacher network reflect its intrinsic composition of semantic and pose information. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student. In this paper, we discuss practical ways to exploit those noisy self-supervision signals with selective transfer for distillation. We further show that self-supervision signals improve conventional distillation with substantial gains under few-shot and noisy-label scenarios. Given the richer knowledge mined from self-supervision, our knowledge distillation approach achieves state-of-the-art performance on standard benchmarks, i.e., CIFAR100 and ImageNet, under both similar-architecture and cross-architecture settings. The advantage is even more pronounced under the cross-architecture setting, where our method outperforms the state of the art CRD by an average of 2.3% in accuracy rate on CIFAR100 across six different teacher-student pairs.

Computer Vision and Pattern Recognition
Teaching What You Should Teach: A Data-Based Distillation Method

Shitong Shao,Huanran Chen,Zhen Huang,Linrui Gong,Shuai Wang,Xinxiao Wu

DOI: https://doi.org/10.48550/arXiv.2212.05422

2022-12-11

Computer Vision and Pattern Recognition

Abstract:In real teaching scenarios, an excellent teacher always teaches what he (or she) is good at but the student is not. This gives the student the best assistance in making up for his (or her) weaknesses and becoming a good one overall. Enlightened by this, we introduce the "Teaching what you Should Teach" strategy into a knowledge distillation framework, and propose a data-based distillation method named "TST" that searches for desirable augmented samples to assist in distilling more efficiently and rationally. To be specific, we design a neural network-based data augmentation module with priori bias, which assists in finding what meets the teacher's strengths but the student's weaknesses, by learning magnitudes and probabilities to generate suitable data samples. By training the data augmentation module and the generalized distillation paradigm in turn, a student model is learned with excellent generalization ability. To verify the effectiveness of our method, we conducted extensive comparative experiments on object recognition, detection, and segmentation tasks. The results on the CIFAR-10, ImageNet-1k, MS-COCO, and Cityscapes datasets demonstrate that our method achieves state-of-the-art performance on almost all teacher-student pairs. Furthermore, we conduct visualization studies to explore what magnitudes and probabilities are needed for the distillation process.
Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

Shoumeng Qiu,Jie Chen,Xinrun Li,Ru Wan,Xiangyang Xue,Jian Pu

2024-07-18

Abstract:In this paper, we introduce a novel knowledge distillation approach for the semantic segmentation task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex teacher models or information from extra sensors. Specifically, for the teacher model training, we propose to noise the label and then incorporate it into input to effectively boost the lightweight teacher performance. To ensure the robustness of the teacher model against the introduced noise, we propose a dual-path consistency training strategy featuring a distance loss between the outputs of two paths. For the student model training, we keep it consistent with the standard distillation for simplicity. Our approach not only boosts the efficacy of knowledge distillation but also increases the flexibility in selecting teacher and student models. To demonstrate the advantages of our Label Assisted Distillation (LAD) method, we conduct extensive experiments on five challenging datasets including Cityscapes, ADE20K, PASCAL-VOC, COCO-Stuff 10K, and COCO-Stuff 164K, five popular models: FCN, PSPNet, DeepLabV3, STDC, and OCRNet, and results show the effectiveness and generalization of our approach. We posit that incorporating labels into the input, as demonstrated in our work, will provide valuable insights into related fields. Code is available at <a class="link-external link-https" href="https://github.com/skyshoumeng/Label_Assisted_Distillation" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Transfer Learning with Label Noise

Xiyu Yu,Tongliang Liu,Mingming Gong,Kun Zhang,Kayhan Batmanghelich,Dacheng Tao

DOI: https://doi.org/10.48550/arxiv.1707.09724

2017-01-01

Abstract:Transfer learning aims to improve learning in target domain by borrowing knowledge from a related but different source domain. To reduce the distribution shift between source and target domains, recent methods have focused on exploring invariant representations that have similar distributions across domains. However, when learning this invariant knowledge, existing methods assume that the labels in source domain are uncontaminated, while in reality, we often have access to source data with noisy labels. In this paper, we first show how label noise adversely affect the learning of invariant representations and the correcting of label shift in various transfer learning scenarios. To reduce the adverse effects, we propose a novel Denoising Conditional Invariant Component (DCIC) framework, which provably ensures (1) extracting invariant representations given examples with noisy labels in source domain and unlabeled examples in target domain; (2) estimating the label distribution in target domain with no bias. Experimental results on both synthetic and real-world data verify the effectiveness of the proposed method.
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher

Guangda Ji,Zhanxing Zhu

DOI: https://doi.org/10.48550/arXiv.2010.10090

2020-10-20

Abstract:Knowledge distillation is a strategy of training a student network with guide of the soft output from a teacher network. It has been a successful method of model compression and knowledge transfer. However, currently knowledge distillation lacks a convincing theoretical understanding. On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency. Based on this metric, we show that for a perfect teacher, a high ratio of teacher's soft labels can be beneficial. Finally, for the case of imperfect teacher, we find that hard labels can correct teacher's wrong prediction, which explains the practice of mixing hard and soft labels.

Machine Learning,Artificial Intelligence

Use of peer role models to increase Pap smear and mammogram screening in Mexican-American and black women.

Learning Student Networks in the Wild

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

DCCD: Reducing Neural Network Redundancy Via Distillation

Remote Sensing Image Scene Classification with Noisy Label Distillation

Evaluation of muscular load of m. biceps brachii in isometric contraction with the use of microvibration.

Learning from a Teacher using Unlabeled Data

Distribution Shift Matters for Knowledge Distillation with Webly Collected Images

Improve Knowledge Distillation via Label Revision and Data Selection

GEIKD : Self-knowledge distillation based on gated ensemble networks and influences-based label noise removal

Small Scale Data-Free Knowledge Distillation

Student as an Inherent Denoiser of Noisy Teacher

Collaborative Knowledge Distillation

Teacher-student collaborative knowledge distillation for image classification

Synthetic data generation method for data-free knowledge distillation in regression neural networks

Fixing the Teacher-Student Knowledge Discrepancy in Distillation

Knowledge Distillation Meets Self-Supervision

Teaching What You Should Teach: A Data-Based Distillation Method

Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

Transfer Learning with Label Noise

Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher