Abstract:Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Ideally, we would like to replace such cumbersome models with simpler models that perform equally well. In this thesis, we study knowledge distillation, the idea of extracting the knowledge contained in a complex model and injecting it into a more convenient model. We present a general framework for knowledge distillation, whereby a convenient model of our choosing learns how to mimic a complex model, by observing the latter's behaviour and being penalized whenever it fails to reproduce it. We develop our framework within the context of three distinct machine learning applications: (a) model compression, where we compress large discriminative models, such as ensembles of neural networks, into models of much smaller size; (b) compact predictive distributions for Bayesian inference, where we distil large bags of MCMC samples into compact predictive distributions in closed form; (c) intractable generative models, where we distil unnormalizable models such as RBMs into tractable models such as NADEs. We contribute to the state of the art with novel techniques and ideas. In model compression, we describe and implement derivative matching, which allows for better distillation when data is scarce. In compact predictive distributions, we introduce online distillation, which allows for significant savings in memory. Finally, in intractable generative models, we show how to use distilled models to robustly estimate intractable quantities of the original model, such as its intractable partition function.

A light-weight edge-enabled knowledge distillation technique for next location prediction of multitude transportation means

Knowledge Distillation for Mobile Edge Computation Offloading

Channel-wise Knowledge Distillation for Dense Prediction

Bayesian-Optimized One-Step Diffusion Model with Knowledge Distillation for Real-Time 3D Human Motion Prediction

Enhancing Mapless Trajectory Prediction through Knowledge Distillation

Real-Time Decentralized knowledge Transfer at the Edge

Leveraging Angular Distributions for Improved Knowledge Distillation

MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction

Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions

A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation

Distillation knowledge-based space-time data prediction on industrial IoT edge devices

Knowledge Distillation with a Precise Teacher and Prediction with Abstention

Distilling Model Knowledge

A Systematic Analysis of Fine-Grained Human Mobility Prediction with On-Device Contextual Data

Cooperative Knowledge-Distillation-Based Tiny DNN for UAV-Assisted Mobile-Edge Network

Efficient Point Cloud Classification via Offline Distillation Framework and Negative-Weight Self-Distillation Technique

Design of a knowledge distillation network for wifi-based indoor localization

A Theoretical Framework Towards Building a Lightweight Model for Pothole Detection using Knowledge Distillation Approach

An Accurate and Low-Parameter Machine Learning Architecture for Next Location Prediction

Enhancing Few-Shot Learning in Lightweight Models via Dual-Faceted Knowledge Distillation

Knowledge Distillation via Weighted Ensemble of Teaching Assistants