Abstract:Most existing approaches often utilize the pre-fixed structure and large number of labeled data for training complex deep models, which are difficult to implement on incremental scenarios. As a matter of fact, real-world data is always in stream form. Thereby, there exits two challenges for building incremental deep models: a) Capacity Scalability. The entire training data is not available before learning the task. It is a challenge to make the deep model structure scale with streaming data for flexible model evolution and faster convergence. b) Capacity Sustainability. The distribution streaming data usually changes in nature (concept drift), thus it is necessary to update the model while preserving previous knowledge for overcoming the catastrophic forgetting. To this end, we develop an incremental deep model (IDM), which expands the network structure according to streaming data and slows down forgetting with the adaptive fisher regularization. However, IDM ignores another significant challenge with streaming data: c) Capacity Demand. Training a deep model always needs a large amount of labeled data, whereas it is almost impossible to label all unlabeled instances in real time. The core problem is to select a small number of the most discriminative instances to label while keeping the predictive accuracy of the model. Thereby, we focus on the online semi-supervised learning scenario with abrupt changes in data distribution, and further improve IDM to a cost-effective incremental deep model (CE-IDM), which can adaptively select the most discriminative newly coming instances for query to reduce the manual labeling costs. Specifically, CE-IDM adopts a novel extensible deep network structure by using an extra attention model for hidden layers. Based on the adaptive attention weights, CE-IDM develops a novel instance selection criterion by jointly estimating unlabeled instances’ representative and informative degree to satisfy the capacity demand. With the newly labeled instances, CE-IDM can quickly update the model with adaptive depth from streaming data and enable capacity scalability. Also, we address capacity sustainability by exploiting the attention based fisher information matrix, which can slow down the forgetting in consequence. Finally, CE-IDM can deal with the three capacity challenges methioned above in a unified framework. We conduct extensive experiments on real-world data and show that CE-IDM outperforms the state-of-the-art methods with a substantial margin.

Incremental Deep Computation Model for Wireless Big Data Feature Learning

An Incremental Deep Convolutional Computation Model for Feature Learning on Industrial Big Data

A Canonical Polyadic Deep Convolutional Computation Model for Big Data Feature Learning in Internet of Things

A Tensor-Train Deep Computation Model for Industry Informatics Big Data Feature Learning

Hierarchical Federated Edge Learning with Adaptive Clustering in Internet of Things

When Wireless Video Streaming Meets AI: A Deep Learning Approach

Deep Reinforcement Learning for Energy Efficiency Maximization in SWIPT-Based Over-the-Air Federated Learning

In-Network Learning Via Over-the-Air Computation in Internet-of-Things

Cost-Effective Incremental Deep Model: Matching Model Capacity with the Least Sampling

Adaptive Deep Models for Incremental Learning

A Deep Residual Computation Model for Heterogeneous Data Learning in Smart Internet of Things

A Tucker Deep Computation Model for Mobile Multimedia Feature Learning

Two-Timescale Digital Twin Assisted Model Interference and Retraining over Wireless Network

Lightweight Tensor Deep Computation Model With Its Application in Intelligent Transportation Systems

Influence of Autoencoder-Based Data Augmentation on Deep Learning-Based Wireless Communication

Dependable Deep Computation Model for Feature Learning on Big Data in Cyber-Physical Systems.

ITCNN: Incremental Learning Network Based on ITDA and Tree Hierarchical CNN.

Class-Incremental Learning for Wireless Device Identification in IoT

Importance-Driven Data Collection for Efficient Online Learning over the Wireless Edge

Deep Computation Model for Unsupervised Feature Learning on Big Data.

A Method for Guaranteeing Wireless Communication Based on a Combination of Deep and Shallow Learning.