A Scheme of Interactive Data Mining Support System in Parallel and Distributed Environment

Zhen Liu,Shinichi Kamohara,Minyi Guo

DOI: https://doi.org/10.1007/3-540-37619-4_27

2003-01-01

Abstract:In this paper, a scheme of interactive data mining support system in high performance parallel and distributed computing environment is proposed. The overall architecture and the mechanism of the system are described.

What problem does this paper attempt to address?

Interactive Data Mining based on PC Cluster Environment

Zhen Liu,Minyi Guo

DOI: https://doi.org/10.1007/0-387-28967-4_16

2005-01-01

Abstract:In order to realize high performance and high effectiveness data mining, not only a parallel and distributed environment, but also an interactive and dynamic visual data mining support are necessary. In this chapter, a scheme of interactive data mining support system in high performance parallel and distributed computing environment is proposed. The overall architecture and the mechanism of the system are described.
A Proposal of High Performance Data Mining System

Zhen Liu,Minyi Guo

DOI: https://doi.org/10.1007/3-540-48051-x_12

2002-01-01

Abstract:In recent years, new decision support system (DSS) based on the technologies of data warehouse, data mining and on-line analytical processing appeared. As the accumulated amount of data becomes enormous too much, the data quantitative problem, the data qualitative problem and the data presentation problem occur in data mining in large-scale databases and data warehouses. An effective way to enhance the power and flexibility of data mining in data warehouses and large databases is to integrate data mining with OLAP in DSS. Parallel and distributed processing are also two important components of successful large-scale data mining applications. In this paper, a high performance data mining scheme is proposed. The overall architecture and the mechanism of the system are described.
A novel agent-based parallel ETL system for massive data

Gang Chen,Baoran An,Yan Liu

DOI: https://doi.org/10.1109/CCDC.2016.7531673

2016-01-01

Abstract:Business Intelligence (BI) systems are crucial for enterprise improvement. They consolidate heterogeneous data from distributed sources and input high-quality data into strategic indicators. An essential component of the data consolidation is Extraction, Transformation and Loading (ETL) which are responsible for extracting data from heterogeneous sources, transforming, restructuring and integrating them into homogenous data warehouse. Due to the deficiency of traditional ETL, the entire ETL component for massive data has decreased performance. Aiming at this challenge, we propose a novel workflow framework for parallel ETL execution based on multi-agent system. The purpose of the system is to utilize a parallel strategy to improve the efficiency of ETL process. Through research, we find that some ETL activities are often executed on the same priority or using the same input data. Based on this discovery, this paper presents a parallel ETL framework based on agent theory and multi-thread techniques. The experimental results show that the proposed approach can greatly improve the efficiency of ETL process.
PDM: A Parallel Data Analysis System Based on Hadoop

DUAN Song-qing,WU Bin,YU Le,WANG Bai

DOI: https://doi.org/10.3969/j.issn.1674-2974.2012.10.015

2012-01-01

Abstract:A PDM(Parallel Data Mining) system was built based on Hadoop.PDM contains a large number of parallel data analysis algorithms based on MapReduce computational framework.These algorithms not only contain the classic algorithms of ETL,data mining,data statistical and text analysis,but also introduce SNA(social network analysis) based on graph mining.The principle and implementation of the parallel multiple linear regression algorithm and the multi-source shortest path algorithm were described and the Message-passing model proposed can effectively solve the problem that MapReduce is difficult to deal with the adjacency matrix structure.This paper also illustrates some typical applications of telecommunications,such as the Business recommendation based on parallel k-means and decision tree algorithms,the Marketing key points discovery based on parallel PageRank algorithm and the like.Finally,the results of performance test show that the proposed system is suitable for dealing with large scale data efficiently.
Distributed Data Mining Based on Multi-agent System

Yan Zhuang,CHEN Ji-Ming,XU Dan,PAN Jin-Gui

2004-01-01

Abstract:The development of data mining is very fast, but most of the technology of the field is for central processing, which require the dataset to collect in a single data warehouse or database for mining. But with the development of network, data nowadays is stored over many machines in different geographical locations. Besides, the amount of data is getting larger and larger, mining such large dataset locally will take us unacceptable time locale memories costs, while this can be improved in the distributed mining systems with plenty compute resource. So it comes out the requirement of Distributed Data Mining(DDM). In DDM, We want to divide the large and complex task into several small and simple ones. Agent supports such ideals with its collaboration and communication. Agents can mining the different datasets locally and combine the partial results into the global result. The article is focus on the Distributed Data Mining Based on Multi-agent system. We will discuss the ideas of Distributed Data Mining and then show the advantages of agents for DDM. Then we propose the problems in the agent-based system for distributed data mining. At last we give some representative agent-based Distributed Data Mining systems.
Distributed Computing Environment: Approaches and Applications

Li Zeng,Lida Xu,Zhongzhi Shi,Maoguang Wang,Wenjuan Wu

DOI: https://doi.org/10.1109/icsmc.2007.4413876

2007-01-01

Abstract:In information systems in particular Business Intelligence systems, for data produced at physically distributed locations most traditional data mining approaches require data to be transmitted to a single location for centralized processing and mining. However, the continual transmission of a large number of data to a central location must be impractical and expensive. Thus, distributed and parallel data mining algorithms and applications were rapidly developed. The paper surveys the-state-of-the art in approaches and applications of distributed computing environment. The goal is to summarize a brief introduction to this field with pointers for further exploration.
Ubiquitous Mining with Interactive Data Mining Agents

Xin-Dong Wu,Xing-Quan Zhu,Qi-Jun Chen,Fei-Yue Wang

DOI: https://doi.org/10.1007/s11390-009-9291-7

IF: 1.871

2009-01-01

Journal of Computer Science and Technology

Abstract:Due to the increasing availability and sophistication of data recording techniques, multiple information sources and distributed computing are becoming the important trends of modern information systems. Many applications such as security informatics and social computing require a ubiquitous data analysis platform so that decisions can be made rapidly under distributed and dynamic system environments. Although data mining has now been popularly used to achieve such goals, building a data mining system is, however, a nontrivial task, which may require a complete understanding on numerous data mining techniques as well as solid programming skills. Employing agent techniques for data analysis thus becomes increasingly important, especially for users not familiar with engineering and computational sciences, to implement an effective ubiquitous mining platform. Such data mining agents should, in practice, be intelligent, complete, and compact. In this paper, we present an interactive data mining agent — OIDM (online interactive data mining), which provides three categories (classification, association analysis, and clustering) of data mining tools, and interacts with the user to facilitate the mining process. The interactive mining is accomplished through interviewing the user about the data mining task to gain efficient and intelligent data mining control. OIDM can help users find appropriate mining algorithms, refine and compare the mining process, and finally achieve the best mining results. Such interactive data mining agent techniques provide alternative solutions to rapidly deploy data mining techniques to broader areas of data intelligence and knowledge informatics.
A hierarchical distributed data mining architecture

Bin Liu,Shu-Gui Cao,Qing-Chun Li,Qi Li

DOI: https://doi.org/10.1109/ICMLC.2011.6016720

2011-01-01

Abstract:Current distributed data mining (DDM) systems popularly assume distributed data sources as partitions of a virtual data table and separately mine them. In fact, when there is essential difference among data sources, the assumption will fail and DDM result quality will also be damaged. For this issue, a hierarchical DDM architecture is proposed by grouping data sources according to their similarity. Ontology technology is adopted to depict the essential content of data sources and measure their similarity.
Data mining in distributed environment: a survey

Gan Wensheng,Lin Jerry Chun-Wei,Chao Han-Chieh,Zhan Justin

DOI: https://doi.org/10.1002/widm.1216

2017-01-01

Abstract:Due to the rapid growth of resource sharing, distributed systems are developed, which can be used to utilize the computations. Data mining (DM) provides powerful techniques for finding meaningful and useful information from a very large amount of data, and has a wide range of real-world applications. However, traditional DM algorithms assume that the data is centrally collected, memory-resident, and static. It is challenging to manage the large-scale data and process them with very limited resources. For example, large amounts of data are quickly produced and stored at multiple locations. It becomes increasingly expensive to centralize them in a single place. Moreover, traditional DM algorithms generally have some problems and challenges, such as memory limits, low processing ability, and inadequate hard disk, and so on. To solve the above problems, DM on distributed computing environment [also called distributed data mining (DDM)] has been emerging as a valuable alternative in many applications. In this study, a survey of state-of-the-art DDM techniques is provided, including distributed frequent item-set mining, distributed frequent sequence mining, distributed frequent graph mining, distributed clustering, and privacy preserving of distributed data mining. We finally summarize the opportunities of data mining tasks in distributed environment. (C) 2017Wiley Periodicals, Inc.
FIU-Miner (a Fast, Integrated, and User-Friendly System for Data Mining) and Its Applications.

Tao Li,Chunqiu Zeng,Wubai Zhou,Wei Xue,Yue Huang,Zheng Liu,Qifeng Zhou,Bin Xia,Qing Wang,Wentao Wang,Xiaolong Zhu

DOI: https://doi.org/10.1007/s10115-016-1014-0

IF: 2.7

2016-01-01

Knowledge and Information Systems

Abstract:The advent of Big Data era drives data analysts from different domains to use data mining techniques for data analysis. However, performing data analysis in a specific domain is not trivial; it often requires complex task configuration, onerous integration of algorithms, and efficient execution in distributed environments. Few efforts have been paid on developing effective tools to facilitate data analysts in conducting complex data analysis tasks. In this paper, we design and implement FIU-Miner, a F ast, I ntegrated, and U ser-friendly system to ease data analysis. FIU-Miner allows users to rapidly configure a complex data analysis task without writing a single line of code. It also helps users conveniently import and integrate different analysis programs . Further, it significantly balances resource utilization and task execution in heterogeneous environments . Case studies of real-world applications demonstrate the efficacy and effectiveness of our proposed system.
Data Mining Support Systems

YL Zhao,JT Yao,YY Yao

DOI: https://doi.org/10.1117/12.543913

2004-01-01

Abstract:The main stream of research in data mining (or knowledge discovery in databases) focuses on algorithms and automatic or semi-automatic processes for discovering knowledge hidden in data. In this paper, we adopt a more general and goal oriented view of data mining. Data mining is regarded as a field of study covering the theories, methodologies, techniques, and activities with the goal of discovering new and useful knowledge. One of its objectives is to design and implement data mining systems. A miner solves problems of data mining manually, or semi-automatically by using such systems. However, there is a lack of studies on how to assist a miner in solving data mining problems. From the experiences and lessons of decision support systems, we introduce the concept of data mining support systems (DMSS). We draw an analogy between the field of decision-making and the field of data mining, and between the role of a manager and the role of a data miner. A DMSS is an active and highly interactive computer system that assists data mining activities. The needs and the basic features of DMSS are discussed.
A System for Parallel Data Mining Service on Cloud

Tao Chen,Jidong Chen,Baoyao Zhou

DOI: https://doi.org/10.1109/CGC.2012.49

2012-01-01

Abstract:We present a cloud-based data mining platform which demonstrates the solution of data mining as a service (DMaaS). In the backend, the data processing engine is based on hadoop, an open-source implementation of Google MapReduce. Implementation of the data mining algorithms in Apache Mahout is deployed in the platform. The user can access DMaaS from his browser for analyzing general purpose data mining problems. In this paper, we give an overview of DMaaS, present the system architecture and implementation techniques, and elaborate on a demonstration scenario.
Distributed data mining for e-business

Bin Liu,Shu Gui Cao,Wu He

DOI: https://doi.org/10.1007/s10799-011-0091-8

2011-01-01

Abstract:In the internet-based e-business environment, most business data are distributed, heterogeneous and private. To achieve true business intelligence, mining large amounts of distributed data is necessary. Through a thorough literature review, this paper identifies four main issues in distributed data mining (DDM) systems for e-business and classifies modern DDM systems into three classes with representative samples. To address these identified issues, this paper proposes a novel DDM model named DRHPDM (Data source Relevance-based Hierarchical Parallel Distributed data mining Model). In addition, to improve the quality of the final result, the data sources are divided into a centralized mining layer and a distributed mining layer, according to their relevance. To improve the openness, cross-platform ability, and intelligence of the DDM system, web service and multi-agent technologies are adopted. The feasibility of DRHPDM was verified by building a prototype system and applying it to a web usage mining scenario.
A Framework to Support Distributed Data Mining on Grid

Kun Gao,Lifeng Xi,Jifang Li

DOI: https://doi.org/10.1007/978-3-540-72434-6_48

2007-01-01

Abstract:In many applications fields, we can obtain benefits from analyzing large distributed data sets by using the high performance computational power. The Grid provides an unrivalled technology for large scale distributed computing as it enables collaboration over the global and the use of distributed computing resources, while also facilitating access to geographically distributed data sets. In this paper, we present a framework for high performance DDM applications in Computational Grid environments called DMGrid, which is based on Grid mechanisms and implemented on top of the Globus 4.0 toolkit.
A Scalable Data Mining Architecture for Bioinformation

R Li,Z Zhang,S Cao,Y Zhu,Y Li

DOI: https://doi.org/10.2495/data030561

2004-01-01

Abstract:Bioinformatics is a new field of data mining research, and data mining is also a promising tool for bioinformatics. Research has been done on how to bridge these two attractive fields. With the critical progression on genetics and high-throughput biotechnologies, all kinds of bio-data have been explosively produced and accumulated. With the success of global views of DNA sequences, gene expression levels, etc. on the genomic scale, now it is possible to discover the nature of life and to promote biology and medicine research further. However, to achieve these goals, accurate large-scale data processing technologies are required. Data mining has been known as a powerful tool for this purpose, but it is still difficult to develop and apply a data mining system for different analysis functions. In this paper, a new scalable data mining architecture for bioinformation analysis, "The Architecture of Bioinformation Data Mining Application Platform", has been proposed to facilitate biologists use of complex data mining technology and to develop a professional data mining system by conveniently developing, customizing or trimming a data mining system for some specific bioresearch. Logically, the architecture is composed of three tiers: a data mining algorithm tier, a analysis logic tier and a profession application tier, with the mapping of data mining algorithm-analysis model-special application as backbone. The architecture can be implemented by using the distributed client/server model. A gene expression data mining system BioMiner is designed and implemented to perform and validate the efficiency of the architecture.
Dynamic Parallel Query Processing for Distributed Objects.

Y Jiang,GR Wang,A Makinouchi

DOI: https://doi.org/10.1109/dexa.1998.707484

1998-01-01

Abstract:An effective exploitation of distributed computing environments to improve the performance of large advanced database applications requires parallel query processing techniques. This is because parallel query execution can be achieved by operations of pipelining and data partitioning, as well as independent subqueries. In order to develop efficient parallel query services in distributed environments, we present a query optimization, a dynamic parallel query scheduling algorithm, and an implementation mechanism for querying distributed objects. Our proposal has four important features.First, the query schedule dynamically schedules independent subqueries in parallel. Second, distributed processors are allowed to execute independent subqueries in random. Third, it doesn't limit the number of processors to deal with parallel query processing in a network-based computing environment. Fourth, it provides simple operations to synchronize parallel query programs. In this paper, we give the query optimization for parallel processing, show an algorithm of dynamic parallel query scheduling, describe an implementation method of the parallel query processing in a network-based distributed computing environment, and present a design of communication and synchronization for the parallel query processing.
OIDM: Online Interactive Data Mining

Qijun Chen,Xindong Wu,Xingquan Zhu

DOI: https://doi.org/10.1007/978-3-540-24677-0_8

2004-01-01

Abstract:Facilitated by the achievements of various data mining techniques, both academic research and industrial applications are using data mining tools to explore knowledge from various databases. However, building a mining system is a nontrivial task, especially for a data mining novice. In this paper, we present an online interactive data mining toolbox - OIDM, which provides three categories (classification, association analysis, and clustering) of data mining tools, and interacts with the user to facilitate his/her mining process. The interactive mining is accomplished through interviewing the user about his/her data mining task. OIDM can help the user find the appropriate mining algorithm, refine the mining process, and finally get the best mining results. To evaluate the system, the website of OIDM (2003) has been released. The feedback is positive. Both students and senior researchers found that OIDM would be useful in conducting data mining research. This research is supported by a NASA EPSCoR grant.
Research on a distributed artificial intelligence and multi-agent system

Huayi Yin,Lizhao Liu,Ying Zhong

2017-01-01

Abstract:This paper examines a distributed artificial intelligence (AI) and multi-agent system (MAS). The data mining (DM) method is proposed to deal with data processing. Integrating DM and MAS can lead to a powerful visualization method for processing large and complex data sets, which represent the kind of easily-accessed information about a business or organization. This method provides a solution for ubiquitous computing, accessing data, and disseminating information to improve the BPs of BPM systems. The graphic representation of challenges and their appropriate solutions is significant in the effective conceptualization of the business context. The modern architectures of a generic AI model and mining algorithms are required for mining complex and large-scale data as well as for describing the various relevant aspects of BP, which in turn, can facilitate the creation of AI systems and tools to maximize user capability. Results show that the use of a distributed AI and MAS in a DM environment can improve the overall system performance.
Research on Distributed Data Mining Technology Based on Grid.

Ning Zhang,Hong Bao

DOI: https://doi.org/10.1109/dbta.2009.106

2009-01-01

Abstract:Distributed data mining (DDM) techniques have become necessary for large and multi-scenario datasets requiring resources, which are heterogeneous and distributed. In this paper, we focused on distributed data mining based in grid. We have discussed and analyzed a new framework based on grid environments to execute new distributed data mining techniques on very large and distributed heterogeneous datasets. The architecture and motivation for the design have been presented. We pointed out that distributed data mining technology can offer a better solution since they are designed to work in a distributed environment by paying careful attention to the computing and communication resources.
High Performance Data Mining Algorithms and Similarity Models Research

Shengjun Xue,Hongtao Wang,Tan Ran

DOI: https://doi.org/10.1109/CSSE.2008.1359

2008-01-01

Abstract:For the complex data of multilevel and large volume distributed in different regions, how to seek and find both be interested and useful information is what scientists are devoted to. Existing efficient method of research is an association rule mining of distributed database system. This paper introduced the distributed association rule mining algorithm. By analyzing the Aproiri algorithm, we have proposed Fast Distributed Mining of Association Rule algorithm and Similarity model. The test results show it can effectively improve the mining accuracy rate.

A Scheme of Interactive Data Mining Support System in Parallel and Distributed Environment

Interactive Data Mining based on PC Cluster Environment

A Proposal of High Performance Data Mining System

A novel agent-based parallel ETL system for massive data

PDM: A Parallel Data Analysis System Based on Hadoop

Distributed Data Mining Based on Multi-agent System

Distributed Computing Environment: Approaches and Applications

Ubiquitous Mining with Interactive Data Mining Agents

A hierarchical distributed data mining architecture

Data mining in distributed environment: a survey

FIU-Miner (a Fast, Integrated, and User-Friendly System for Data Mining) and Its Applications.

Data Mining Support Systems

A System for Parallel Data Mining Service on Cloud

Distributed data mining for e-business

A Framework to Support Distributed Data Mining on Grid

A Scalable Data Mining Architecture for Bioinformation

Dynamic Parallel Query Processing for Distributed Objects.

OIDM: Online Interactive Data Mining

Research on a distributed artificial intelligence and multi-agent system

Research on Distributed Data Mining Technology Based on Grid.

High Performance Data Mining Algorithms and Similarity Models Research