Abstract:In the era of big data, practical applications in various domains continually generate large-scale time-series data. Among them, some data show significant or potential periodicity characteristics, such as meteorological and financial data. It is critical to efficiently identify the potential periodic patterns from massive time-series data and provide accurate predictions. In this paper, a Periodicity-based Parallel Time Series Prediction (PPTSP) algorithm for large-scale time-series data is proposed and implemented in the Apache Spark cloud computing environment. To effectively handle the massive historical datasets, a Time Series Data Compression and Abstraction (TSDCA) algorithm is presented, which can reduce the data scale as well as accurately extracting the characteristics. Based on this, we propose a multi-layer time series periodic pattern recognition (MTSPPR) algorithm using the Fourier Spectrum Analysis (FSA) method. In addition, a Periodicity-based Time Series Prediction (PTSP) algorithm is proposed. Data in the subsequent period are predicted based on all previous period models, in which a time attenuation factor is introduced to control the impact of different periods on the prediction results. Moreover, to improve the performance of the proposed algorithms, we propose a parallel solution on the Apache Spark platform, using the Streaming real-time computing module. To efficiently process the large-scale time-series datasets in distributed computing environments, Distributed Streams (DStreams) and Resilient Distributed Datasets (RDDs) are used to store and calculate these datasets. Logical and data dependencies of RDDs in the P-TSDCA, P-MTSPPR, and P-PTSP processes are considered, and the corresponding parallel execution solutions are conducted. Extensive experimental results show that our PPTSP algorithm has significant advantages compared with other algorithms in terms of prediction accuracy and performance.

SPM: Modeling Spark Task Execution Time from the Sub-stage Perspective.

Improving MapReduce Performance with Partial Speculative Execution

Fixed-point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Optimizing Resource Allocation for Data-Parallel Jobs Via GCN-Based Prediction

Real-time scheduling for tasks with uncertain execution times

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

Learning-based SPARQL Query Performance Modeling and Prediction

The Tiny-Tasks Granularity Trade-Off: Balancing overhead vs. performance in parallel systems

TR-Spark

Reliable Estimation of Execution Time of MapReduce Program

Optimizing Assembly Sequence Planning Using Precedence Graph-Based Assembly Subsets Prediction Method

Performance Prediction for Large-Scale Parallel Applications Using Representative Replay

Dynamic Job Shop Scheduling Based on Order Remaining Completion Time Prediction

Grid scheduling based on prediction of task completion time

A Real-Time Partition Generation Mechanism for Data Skew Mitigation in Spark Computing Environment

Predicting Execution Time of Computer Programs Using Sparse Polynomial Regression.

An Optimal Locality-Aware Task Scheduling Algorithm Based on Bipartite Graph Modelling for Spark Applications

SP-Partitioner: A novel partition method to handle intermediate data skew in spark streaming

SPM: Sparse Persistent Memory Attention-Based Model for Network Traffic Prediction

InSTechEM: An Internet of Thing big data–oriented extended model for MapReduce performance prediction in multiple edge clouds

A periodicity-based parallel time series prediction algorithm in cloud computing environments