Abstract:Due to the rapid development of information technology and network technology, there is a lot of data, but the phenomenon of lack of knowledge is becoming more and more serious. Data mining technology has developed vigorously in this environment, and it has shown more and more vitality. Based on Spark programming model, this paper designs the parallel extension of fuzzy c-means. In order to enhance the performance of fuzzy c-means parallel expansion, the improvement strategy of k-means during the initialization phase is borrowed, and k-means// is extended to fuzzy c-means to obtain better clustering performance. Combined with Spark's programming model, this paper can obtain extended parallel fuzzy c-means algorithm. Several experiments on the data set of the algorithm proposed in this paper have shown good scalability and parallelism, effectively expanding fuzzy c-means clustering to distributed applications, greatly increasing the scale of the data processed by the algorithm. This improves the robustness of the algorithm and the adaptability of the algorithm to the shape and structure of the data, so that the parallel and scalable clustering algorithm can more effectively perform cluster analysis on big data. Three algorithms were simulated on MATLAB platform. We use simple data sets and complex two-dimensional data sets, and compare with the traditional fuzzy c-means algorithm and fuzzy c-means algorithm based on fuzzy entropy. Experiments show that the scalable parallel fuzzy c-means algorithm not only greatly improves the anti-noise performance, but also improves the convergence speed, and it can automatically determine the optimal number of clusters.

The Parallel Implementation of Deep Neural Decision Tree Based on Spark

Parallelization of Classification Algorithms Based on SparkR

A Communication-Efficient Parallel Algorithm for Decision Tree

Decision Stream: Cultivating Deep Decision Trees

Single MCMC Chain Parallelisation on Decision Trees

Learn Decision Trees with Deep Visual Primitives

Design and Implementation of Parallel DBSCAN Algorithm Based on Spark

A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment

Parallelization of Machine Learning Algorithms Respectively on Single Machine and Spark

Study of ELM Algorithm Parallelization Based on Spark

Parallel naive Bayes algorithm for large-scale Chinese text classification based on spark

Dive into Decision Trees and Forests: A Theoretical Demonstration

A generalized decision tree ensemble based on the NeuralNetworks architecture: Distributed Gradient Boosting Forest (DGBF)

Traffic Flow Prediction with Convolutional Neural Network Accelerated by Spark Distributed Cluster

Traffic Network Flow Prediction Using Parallel Training for Deep Convolutional Neural Networks on Spark Cloud

The parallel algorithms for LIBSVM parameter optimization based on Spark

A parallel fuzzy rule-base based decision tree in the framework of map-reduce

Distilling a Neural Network Into a Soft Decision Tree

Data Mining Algorithm for Cloud Network Information Based on Artificial Intelligence Decision Mechanism

A Deep Random Forest Model on Spark for Network Intrusion Detection