Abstract:In recent years, with rapid economic development, air pollution has become extremely serious, causing many negative effects on health, environment and medical costs. PM2.5 is one of the main components of air pollution. Therefore, it is necessary to know the PM2.5 air quality in advance for health. Many studies on air quality are based on the government’s official air quality monitoring stations, which cannot be widely deployed due to high cost constraints. Furthermore, the update frequency of government monitoring stations is once an hour, and it is hard to capture short-term PM2.5 concentration peaks with little warning. Nevertheless, dealing with short-term data with many stations, the volume of data is huge and is calculated, analyzed and predicted in a complex way. This alleviates the high computational requirements of the original predictor, thus making Spark suitable for the considered problem. This study proposes a PM2.5 instant prediction architecture based on the Spark big data framework to handle the huge data from the LASS community. The Spark big data framework proposed in this study is divided into three modules. It collects real time PM2.5 data and performs ensemble learning through three machine learning algorithms (Linear Regression, Random Forest, Gradient Boosting Decision Tree) to predict the PM2.5 concentration value in the next 30 to 180 min with accompanying visualization graph. The experimental results show that our proposed Spark big data ensemble prediction model in next 30-min prediction has the best performance (R2 up to 0.96), and the ensemble model has better performance than any single machine learning model. Taiwan has been suffering from a situation of relatively poor air pollution quality for a long time. Air pollutant monitoring data from LASS community can provide a wide broader monitoring, however the data is large and difficult to integrate or analyze. The proposed Spark big data framework system can provide short-term PM2.5 forecasts and help the decision-maker to take proper action immediately.

Efficient inference of large-scale air quality using a lightweight ensemble predictor

Spatially Fine-Grained Urban Air Quality Estimation Using Ensemble Semi-Supervised Learning and Pruning.

Predicting Air Quality Using a Multi-Scale Spatiotemporal Graph Attention Network

A deep spatio-temporal learning network for continuous citywide air quality forecast based on dense monitoring data

A Deep Spatial-Temporal Ensemble Model for Air Quality Prediction

Large Scale Air Pollution Prediction with Deep Convolutional Networks

Forecasting PM2.5 Using Hybrid Graph Convolution-Based Model Considering Dynamic Wind-Field to Offer the Benefit of Spatial Interpretability.

Prediction of PM2.5 Concentration Based on Deep Learning, Multi-Objective Optimization, and Ensemble Forecast

Spatiotemporal hierarchical transmit neural network for regional-level air-quality prediction

Deep Distributed Fusion Network for Air Quality Prediction

Design of a Spark Big Data Framework for PM2.5 Air Pollution Forecasting

Deep Inferential Spatial-Temporal Network for Forecasting Air Pollution Concentrations

Temporal-Spatial Aggregated Urban Air Quality Inference With Heterogeneous Big Data

Air Pollution Concentration Forecast Method Based on the Deep Ensemble Neural Network

A Model-driven and Data-driven Fusion Framework for Accurate Air Quality Prediction

Ensemble Multitask Prediction of Air Pollutants Time Series: Based on Variational Inference, Data Projection, and Generative Adversarial Network

Predicting Fine-Grained Air Quality Based on Deep Neural Networks

An adaptive adjacency matrix-based graph convolutional recurrent network for air quality prediction

Time-sensitive prediction of NO2 concentration in China using an ensemble machine learning model from multi-source data

Pm2.5 Air Quality Index Prediction Using An Ensemble Learning Model

Regional PM2.5 Prediction with Hybrid Directed Graph Neural Networks and Spatio-temporal Fusion of Meteorological Factors