Abstract:Educational big data analysis is facilitated by the significant amount of unstructured data found in education institutions. Python has various toolkits for both structured and unstructured data processing. However, its ability for processing large-scale data is limited. On the other hand, Spark is a big data processing framework, but it does not have the needed toolkits for processing unstructured rich text documents, 3D model and image processing. In this study, we develop a generic framework that integrates Python toolkits and Spark based on service-oriented architecture. The framework automatically extends the serial algorithm written in Python to distributed algorithm to accomplish parallel processing tasks seamlessly. First, our focus is on achieving non-intrusive deployment to Spark servers and how to run Python codes in Spark environment to process rich text documents. Second, we propose a compression-based schema to address the poor performance of small sized files in HDFS. Finally, we design a generic model that can process different types of poly-structured data such as 3D models and images. We published the services used in the system for sharing them at https level for constructing different systems. It is evaluated through simulation experiments using large-scale rich text documents, 3D models and images. According to the results, the speedup is 49 times faster than the standalone Python-docx in the simulations of extracting 232 GB docx files when eight physical nodes with 128 cores are used. It reaches about 89 times after further compression schema is applied. In addition, simulations for 3D model descriptors' extraction show that the simulation achieves a speedup of about 116 times. In the large-scale image's HOG features extraction simulation task of up to 256.7 GB (6,861,024 images), a speedup of up to 110 times is achieved.

Octopus-DF: Unified DataFrame-based cross-platform data analytic system

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

Unified Programming Model and Software Framework for Big Data Machine Learning and Data Analytics.

Distributed High-Dimension Matrix Operation Optimization on Spark

Distributed matrix computing system for big data

Supercharging Distributed Computing Environments For High Performance Data Engineering

Supercharging distributed computing environments for high-performance data engineering

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

In-depth Analysis On Parallel Processing Patterns for High-Performance Dataframes

A Service-Oriented Framework for Large-Scale Documents Processing and Application via 3D Models and Feature Extraction

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

Xorbits: Automating Operator Tiling for Distributed Data Science

SparkDQ: Efficient Generic Big Data Quality Management on Distributed Data-Parallel Computation

Octopus: Experiences with a Hybrid Event-Driven Architecture for Distributed Scientific Computing

A Time-Cost Based Automatic Scheduling Framework for Matrix Computation on Various Distributed Computing Platforms

A Dwarf-based Scalable Big Data Benchmarking Methodology

DFS-Perf : A Scalable and Unified Benchmarking Framework for Distributed File Systems

AFrame: Extending DataFrames for Large-Scale Modern Data Analysis (Extended Version)

Epic: an Extensible and Scalable System for Processing Big Data.

SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile

NO2: Speeding Up Parallel Processing of Massive Compute-Intensive Tasks