Abstract:In the big data era, characterized by vast volumes of complex data, the efficiency of machine learning models is of utmost importance, particularly in the context of intelligent agriculture. Knowledge distillation (KD), a technique aimed at both model compression and performance enhancement, serves as a pivotal solution by distilling the knowledge from an elaborate model (teacher) to a lightweight, compact counterpart (student). However, the true potential of KD has not been fully explored. Existing approaches primarily focus on transferring instance-level information by big data technologies, overlooking the valuable information embedded in token-level relationships, which may be particularly affected by the long-tail effects. To address the above limitations, we propose a novel method called Knowledge Distillation with Token-level Relationship Graph (TRG) that leverages token-wise relationships to enhance the performance of knowledge distillation. By employing TRG, the student model can effectively emulate higher-level semantic information from the teacher model, resulting in improved performance and mobile-friendly efficiency. To further enhance the learning process, we introduce a dynamic temperature adjustment strategy, which encourages the student model to capture the topology structure of the teacher model more effectively. We conduct experiments to evaluate the effectiveness of the proposed method against several state-of-the-art approaches. Empirical results demonstrate the superiority of TRG across various visual tasks, including those involving imbalanced data. Our method consistently outperforms the existing baselines, establishing a new state-of-the-art performance in the field of KD based on big data technologies.

Knowledge Distillation for Travel Time Estimation

Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion

When Will You Arrive? Estimating Travel Time Based on Deep Neural Networks

TADNM: A Transportation-Mode Aware Deep Neural Model for Travel Time Estimation.

AttentionTTE: a deep learning model for estimated time of arrival

Knowledge Distillation on Spatial-Temporal Graph Convolutional Network for Traffic Prediction

Fine-Grained Trajectory-based Travel Time Estimation for Multi-city Scenarios Based on Deep Meta-Learning

MCT-TTE: Travel Time Estimation Based on Transformer and Convolution Neural Networks

Package Arrival Time Prediction via Knowledge Distillation Graph Neural Network

Travel Time Estimation by Learning Driving Habits and Traffic Conditions

Highlight Every Step: Knowledge Distillation via Collaborative Teaching

DutyTTE: Deciphering Uncertainty in Origin-Destination Travel Time Estimation

A Temporal Convolutional Knowledge Tracing Model Integrating Forgetting Factors and Item Response Theory

TTPNet: A Neural Network for Travel Time Prediction Based on Tensor Decomposition and Graph Embedding

DeepTravel: a Neural Network Based Travel Time Estimation Model with Auxiliary Supervision

TransETA: transformer networks for estimated time of arrival with local congestion representation

Knowledge Distillation via Token-Level Relationship Graph Based on the Big Data Technologies

Teacher-student collaborative knowledge distillation for image classification

Enhancing Mapless Trajectory Prediction through Knowledge Distillation

Ensemble Knowledge Distillation for CTR Prediction