Abstract:In recent years, with rapid economic development, air pollution has become extremely serious, causing many negative effects on health, environment and medical costs. PM2.5 is one of the main components of air pollution. Therefore, it is necessary to know the PM2.5 air quality in advance for health. Many studies on air quality are based on the government’s official air quality monitoring stations, which cannot be widely deployed due to high cost constraints. Furthermore, the update frequency of government monitoring stations is once an hour, and it is hard to capture short-term PM2.5 concentration peaks with little warning. Nevertheless, dealing with short-term data with many stations, the volume of data is huge and is calculated, analyzed and predicted in a complex way. This alleviates the high computational requirements of the original predictor, thus making Spark suitable for the considered problem. This study proposes a PM2.5 instant prediction architecture based on the Spark big data framework to handle the huge data from the LASS community. The Spark big data framework proposed in this study is divided into three modules. It collects real time PM2.5 data and performs ensemble learning through three machine learning algorithms (Linear Regression, Random Forest, Gradient Boosting Decision Tree) to predict the PM2.5 concentration value in the next 30 to 180 min with accompanying visualization graph. The experimental results show that our proposed Spark big data ensemble prediction model in next 30-min prediction has the best performance (R2 up to 0.96), and the ensemble model has better performance than any single machine learning model. Taiwan has been suffering from a situation of relatively poor air pollution quality for a long time. Air pollutant monitoring data from LASS community can provide a wide broader monitoring, however the data is large and difficult to integrate or analyze. The proposed Spark big data framework system can provide short-term PM2.5 forecasts and help the decision-maker to take proper action immediately.

Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data on spark distributed framework

Predicting Air Quality Using a Multi-Scale Spatiotemporal Graph Attention Network

Design of a Spark Big Data Framework for PM2.5 Air Pollution Forecasting

Efficient Spatiotemporal Interpolation with Spark Machine Learning

Distributed Streaming Analytics on Large-scale Oceanographic Data using Apache Spark

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Spatial overlay analysis of land use vector data based on Spark

Spatiotemporal hierarchical transmit neural network for regional-level air-quality prediction

High spatio-temporal resolution predictions of PM 2.5 using low-cost sensor data

A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment

A Spatiotemporal Prediction Framework for Air Pollution Based on Deep RNN

Fine-Grained Air Quality Inference with Remote Sensing Data and Ubiquitous Urban Data

Data-driven spatial-temporal prediction of electric vehicle load profile considering charging behavior

Forecasting urban air pollution using multi-site spatiotemporal data fusion method (Geo-BiLSTMA)

Simulation of Temporal and Spatial Distribution Characteristics of Air Pollutant Concentration in Residential Areas Based on Random Forest

A Novel Random Forest and its Application on Classification of Air Quality

A scalable and real-time system for disease prediction using big data processing

Street-level Air Quality Inference Based on Geographically Context-aware Random Forest Using Opportunistic Mobile Sensor Network

Imputing Satellite-Derived Aerosol Optical Depth Using a Multi-Resolution Spatial Model and Random Forest for PM2.5 Prediction

Optimizing and accelerating space-time Ripley's K function based on Apache Spark for distributed spatiotemporal point pattern analysis