MODL: Multilearner Online Deep Learning

Antonios Valkanas,Boris N. Oreshkin,Mark Coates

2024-05-28

Abstract:Online deep learning solves the problem of learning from streams of data, reconciling two opposing objectives: learn fast and learn deep. Existing work focuses almost exclusively on exploring pure deep learning solutions, which are much better suited to handle the "deep" than the "fast" part of the online learning equation. In our work, we propose a different paradigm, based on a hybrid multilearner approach. First, we develop a fast online logistic regression learner. This learner does not rely on backpropagation. Instead, it uses closed form recursive updates of model parameters, handling the fast learning part of the online learning problem. We then analyze the existing online deep learning theory and show that the widespread ODL approach, currently operating at complexity $O(L^2)$ in terms of the number of layers $L$, can be equivalently implemented in $O(L)$ complexity. This further leads us to the cascaded multilearner design, in which multiple shallow and deep learners are co-trained to solve the online learning problem in a cooperative, synergistic fashion. We show that this approach achieves state-of-the-art results on common online learning datasets, while also being able to handle missing features gracefully. Our code is publicly available at <a class="link-external link-https" href="https://github.com/AntonValk/MODL" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper mainly explores the problem of online deep learning (ODL), especially the challenge of learning quickly and effectively when dealing with data streams. Existing work mainly focuses on pure deep learning solutions, which may not be as efficient in terms of "fast" as they are in terms of "deep". The authors propose a new framework called Multiple Learners Online Deep Learning (MODL), which is based on a hybrid multiple learners approach and aims to address the contradiction between these two goals.

The core ideas of MODL include:
1. Parallel learning: Reduce the dependence between learners so that each learner can update parameters independently and quickly.
2. Efficient statistical approximation: Some learners can use closed-form recursive updates to learn quickly.
3. Cooperative learning: Combine the outputs of learners in a delta mode to enable them to work together to solve problems.

In the paper, the authors develop a fast online logistic regression learner that does not rely on backpropagation, and analyze the existing ODL theory, showing that its complexity can be reduced from O(L^2) to O(L). In this way, they design a cascaded multiple learner structure where shallow and deep learners are jointly trained to solve online learning problems.

Experimental results show that MODL achieves state-of-the-art results on standard online learning datasets and can handle missing features gracefully. In addition, compared to existing ODL methods, MODL has lower training complexity.

Overall, this paper attempts to address how to achieve fast and deep online learning when dealing with data streams. By proposing the MODL framework, it improves learning efficiency and accuracy.