Automated data processing and feature engineering for deep learning and big data applications: a survey

Alhassan Mumuni,Fuseini Mumuni

DOI: https://doi.org/10.1016/j.jiixd.2024.01.002

2024-03-19

Abstract:Modern approach to artificial intelligence (AI) aims to design algorithms that learn directly from data. This approach has achieved impressive results and has contributed significantly to the progress of AI, particularly in the sphere of supervised deep learning. It has also simplified the design of machine learning systems as the learning process is highly automated. However, not all data processing tasks in conventional deep learning pipelines have been automated. In most cases data has to be manually collected, preprocessed and further extended through data augmentation before they can be effective for training. Recently, special techniques for automating these tasks have emerged. The automation of data processing tasks is driven by the need to utilize large volumes of complex, heterogeneous data for machine learning and big data applications. Today, end-to-end automated data processing systems based on automated machine learning (AutoML) techniques are capable of taking raw data and transforming them into useful features for Big Data tasks by automating all intermediate processing stages. In this work, we present a thorough review of approaches for automating data processing tasks in deep learning pipelines, including automated data preprocessing--e.g., data cleaning, labeling, missing data imputation, and categorical data encoding--as well as data augmentation (including synthetic data generation using generative AI methods) and feature engineering--specifically, automated feature extraction, feature construction and feature selection. In addition to automating specific data processing tasks, we discuss the use of AutoML methods and tools to simultaneously optimize all stages of the machine learning pipeline.

Machine Learning,Artificial Intelligence,Databases

What problem does this paper attempt to address?

The main goal of this paper is to address the automation of data processing tasks in deep learning and big data applications. Specifically, the authors focus on the automation of data preprocessing (including data cleaning, labeling, missing value imputation, categorical encoding, etc.) and feature engineering (including feature extraction, construction, and selection). Although modern deep learning methods have largely achieved the ability to automatically extract useful features from raw data, real-world data often have quality issues such as noise, missing values, or skewness, which limit the model's performance. Therefore, the paper aims to explore how to improve these data processing tasks through automation, thereby enhancing the performance and reliability of the final machine learning model.

In addition, the paper discusses data augmentation techniques, which involve transforming existing data to generate new training samples, increasing the quantity and diversity of the dataset, and thus improving the model's generalization ability. This process can also be automated and is very effective in addressing issues such as data imbalance and data bias.

In summary, this paper aims to simplify the entire data processing workflow through comprehensive automation methods, allowing developers to avoid manually performing tedious data preparation and feature engineering tasks, thereby enabling machine learning models to better handle complex and large-scale data challenges.