Abstract:Many distributed machine learning frameworks have recently been built to speed up the large-scale data learning process. However, most distributed machine learning used in these frameworks still uses an offline algorithm model which cannot cope with the data stream problems. In fact, large-scale data are mostly generated by the non-stationary data stream where its pattern evolves over time. To address this problem, we propose a novel Evolving Large-scale Data Stream Analytics framework based on a Scalable Parsimonious Network based on Fuzzy Inference System (Scalable PANFIS), where the PANFIS evolving algorithm is distributed over the worker nodes in the cloud to learn large-scale data stream. Scalable PANFIS framework incorporates the active learning (AL) strategy and two model fusion methods. The AL accelerates the distributed learning process to generate an initial evolving large-scale data stream model (initial model), whereas the two model fusion methods aggregate an initial model to generate the final model. The final model represents the update of current large-scale data knowledge which can be used to infer future data. Extensive experiments on this framework are validated by measuring the accuracy and running time of four combinations of Scalable PANFIS and other Spark-based built in algorithms. The results indicate that Scalable PANFIS with AL improves the training time to be almost two times faster than Scalable PANFIS without AL. The results also show both rule merging and the voting mechanisms yield similar accuracy in general among Scalable PANFIS algorithms and they are generally better than Spark-based algorithms. In terms of running time, the Scalable PANFIS training time outperforms all Spark-based algorithms when classifying numerous benchmark datasets.

Scikit-Multiflow: A Multi-output Streaming Framework

StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences

Standardized Evaluation of Machine Learning Methods for Evolving Data Streams

A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning

Reasonable Scale Machine Learning with Open-Source Metaflow

A Framework of Online Learning with Imbalanced Streaming Data.

FLUID: A Unified Evaluation Framework for Flexible Sequential Data

Online feature selection for multi-source streaming features

MORStreaming: A Multioutput Regression System for Streaming Data

stream-learn -- open-source Python library for difficult data stream batch analysis

Aequitas Flow: Streamlining Fair ML Experimentation

scikit-dyn2sel -- A Dynamic Selection Framework for Data Streams

Multi-Source Transfer Learning for Non-Stationary Environments

FastFlow: Efficient Scalable Model-Driven Framework for Processing Massive Mobile Stream Data.

A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework

STREAMLINE: Streaming Active Learning for Realistic Multi-Distributional Settings

MSFlow: Multi-Scale Flow-based Framework for Unsupervised Anomaly Detection

STREAMLINE: A Simple, Transparent, End-To-End Automated Machine Learning Pipeline Facilitating Data Analysis and Algorithm Comparison

Evolving Large-Scale Data Stream Analytics based on Scalable PANFIS

Multisensor-Pipeline: A Lightweight, Flexible, and Extensible Framework for Building Multimodal-Multisensor Interfaces

A Scalable Software Framework for Stateful Stream Data Processing on Multiple GPUs and Applications