Abstract:Limited by the scale and diversity of time series data, the neural networks trained on time series data often overfit and show unsatisfacotry performances. In comparison, large language models (LLMs) recently exhibit impressive generalization in diverse fields. Although massive LLM based approaches are proposed for time series tasks, these methods require to load the whole LLM in both training and reference. This high computational demands limit practical applications in resource-constrained settings, like edge-computing and IoT devices. To address this issue, we propose Knowledge Pruning (KP), a novel paradigm for time series learning in this paper. For a specific downstream task, we argue that the world knowledge learned by LLMs is much redundant and only the related knowledge termed as "pertinent knowledge" is useful. Unlike other methods, our KP targets to prune the redundant knowledge and only distill the pertinent knowledge into the target model. This reduces model size and computational costs significantly. Additionally, different from existing LLM based approaches, our KP does not require to load the LLM in the process of training and testing, further easing computational burdens. With our proposed KP, a lightweight network can effectively learn the pertinent knowledge, achieving satisfactory performances with a low computation cost. To verify the effectiveness of our KP, two fundamental tasks on edge-computing devices are investigated in our experiments, where eight diverse environments or benchmarks with different networks are used to verify the generalization of our KP. Through experiments, our KP demonstrates effective learning of pertinent knowledge, achieving notable performance improvements in regression (19.7% on average) and classification (up to 13.7%) tasks, showcasing state-of-the-art results.

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve This paper aims to address the issues encountered when analyzing time series data on edge computing devices. Specifically, existing methods face the following challenges when dealing with time series data: 1. **Overfitting and Poor Performance**: Due to the limited scale and diversity of time series data, the trained neural networks often tend to overfit and perform poorly. 2. **Computational Resource Constraints**: Although methods based on large language models (LLM) perform well in time series tasks, these methods require loading the entire LLM during training and inference, leading to high computational demands, which limits their practical application in resource-constrained environments (such as edge computing and IoT devices). To address these issues, the authors propose a new compression paradigm—Knowledge Pruning (KP). The main objectives of KP are: - **Identifying and Pruning Redundant Knowledge**: For specific downstream tasks, a large amount of world knowledge contained in LLMs is actually redundant, and only the task-relevant portion of the knowledge (referred to as "relevant knowledge") is useful. KP effectively transfers the relevant knowledge to the target model by pruning the redundant knowledge. - **Reducing Model Size and Computational Cost**: Through the above process, KP significantly reduces the model size and computational cost, enabling lightweight networks to achieve satisfactory performance at low computational costs. Experimental validation shows that KP achieves significant performance improvements in both regression and classification tasks, with an average increase of 19.7% and up to 13.7%, respectively, demonstrating state-of-the-art results.

LLM-based Knowledge Pruning for Time Series Data Analytics on Edge-computing Devices

Class-Aware Pruning for Efficient Neural Networks

Enabling Deep Learning on Edge Devices through Filter Pruning and Knowledge Transfer

Enhancing Scalability in Recommender Systems through Lottery Ticket Hypothesis and Knowledge Distillation-based Neural Network Pruning

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

Pruning as a Domain-specific LLM Extractor

Pruning Foundation Models for High Accuracy without Retraining

lAKECP: Adaptive Knowledge Extraction from Feature Maps for Fast and Efficient Channel Pruning

SparseLLM: Towards Global Pruning for Pre-trained Language Models

KDnet-RUL: A Knowledge Distillation Framework to Compress Deep Neural Networks for Machine Remaining Useful Life Prediction

Cloud–Edge Collaborative Inference with Network Pruning

Edge-enabled anomaly detection and information completion for social network knowledge graphs

Large Language Model Pruning

LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

Intermediate-grained kernel elements pruning with structured sparsity

LNPT: Label-free Network Pruning and Training

Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments

Data-Efficient Adaptive Global Pruning for Convolutional Neural Networks in Edge Computing

LSOP: Layer-Scaled One-shot Pruning

Model Selection - Knowledge Distillation Framework for Model Compression