Data Preprocessing of Spatial Data Mining

Hanning Yuan,Shuliang Wang

DOI: https://doi.org/10.1117/12.650728

2005-01-01

Abstract:Data preprocessing is essential in spatial data mining. This paper presents some issues on data preprocessing. It pays more attention on incomplete data, inaccurate data, repetitive data, inconsistent data, and image data. Finally, a case is studied, and come with a satisfactory result.

What problem does this paper attempt to address?

Research on Data Preprocessing Methods for Big Data

Qin KONG,Chang-qing YE,Yun SUN

DOI: https://doi.org/10.3969/j.issn.1673-629X.2018.05.001

2018-01-01

Abstract:In the era of big data,it is an enormous challenge about data perception,expression,understanding and computing due to the in-herent complexity of data type,organization pattern,different relations and data quality.Data preprocessing is a very important preparation before data analysis and mining.On the one hand,it ensures the correctness and effectiveness of data mining.On the other hand,the ad-justment of the data format and content makes data meet the demand of mining.We analyze the main tasks of data preprocessing and sum-marize several popular processing methods for handling various kinds of"dirty data".The algorithms of data cleaning,integration,trans-formation and reduction are discussed in detail.Using such kinds of preprocessing methods,we can remove redundant and error data,im-prove the incomplete data,promote the required data integration,help data refinement and data consistency of centralized storage.We also can get the minimum and the most reliable data set necessary for the mining system.It also reduces the cost of data mining and improves the accuracy,validity and practicability of knowledge discovery.
A Combined Data Preprocessing Method Based on K-means Clustering and Singular Spectrum Analysis

Ya Yao,Baoliang Wang,Zhiyao Huang,Haifeng Ji,Haiqing Li

DOI: https://doi.org/10.1109/icetce.2012.7

2012-01-01

Abstract:The original data collected by the online monitoring system are often cluttered and redundant, which result in the low performance of water quality prediction. A data preprocessing method is introduced in this research to solve the problem. There are 2 steps in this method. Firstly, k-means clustering is used to classify monitoring data into several clusters. Each of the clusters has a representative data value. The results can be considered containing all the information of water quality of a specific period of time. Secondly, singular spectrum analysis (SSA) is used to capture significant component of classified data of the specific period of time. To evaluate the performance of the data preprocess method, the back propagation artificial neural networks(BP ANN) is introduced as the prediction model. The preprocessed data of the specific period of time is set as the input of BP ANN, while the output data is the forecast result. The result is compared with prediction without data preprocessing algorithm. It shows that the prediction with data preprocessing method performs better.
Data Mining In Multisensor System Based On Rough Set Theory

Bin Han,Tiejun Wu

DOI: https://doi.org/10.1109/ACC.2001.945675

2001-01-01

Abstract:In this paper we discuss how data fusion and data mining are mutually supportive in multisensor systems and, with an illustrative example, we show Rough Set theory is a powerful tool to inducing rule-based object models for a data fusion system used in environment pollution monitoring. The entropy theory is employed to analyze the attribute sets. Also Rough probability model is introduced to mine the statistical information hidden in the information table. They all show promising future in processing the data coming from the data fusion system.
QUALITY ASSESSMENT AND UNCERTAINTY HANDLING IN SPATIAL DATA MINING

Binbin He,Dazhi Guo

2004-01-01

Abstract:Spatial data mining refers to extracting or "mining" knowledge or interesting spatial / non- spatial patterns (rules) hidden in spatial databases, and usually the data in databases are characterized by large-amounts, incompleteness, noise, fuzziness and randomness. The most researches of spatial data mining focused on the method of data mining and its algorithms in the past years. Although uncertainties exist in spatial data mining, they had not been paid much attention to. The overall objective in this paper is to develop a framework for quality assessment and uncertainty handling in spatial data mining. For this purpose, we have adopted several quality assessment indices for the results of spatial data mining and a methodology for handling uncertainty in spatial data mining based on fuzzy logic and Dempster-Shafer theory. Firstly, the uncertainties at various stages of spatial data mining are briefly analyzed. And some kinds of quality assessment indices such as data clustering, classification and association rule mining are adopted. Then, the uncertainty in spatial data mining is probed. In this connection, fuzzy logic and Dempster- Shafer theory are used for the representation and handling of uncertainty in the process of spatial data mining.
The Technology and Methods of Spatial Data Mining and Their Application

田庆久,毛克彪

DOI: https://doi.org/10.3969/j.issn.1004-0323.2002.04.005

2002-01-01

Abstract:Remotely sensed data are so large that they can not be dealed with by human brains. Therefore, it becomes increasely important to find a method to get information automatically, quickly, efficiently from remotely sensed data. The technology of Data Mining was put up with in 1980's, and it has been used in many fields, such as finance fields, insurance fields, retail fields, medical and justice, etc. Data mining is the non-trivial process of identifying valid, novel, potentially useful, ultimately understandable patterns from huge volume of data. The spatial data mining is developed on the base of it. In a sense, spatial data mining resolves the problem of abundant data that human brain cannot be competent with. Furthermore, it can be used in artificial intelligence and pattern recognition fields. Spatial data mining differentiates from general data mining. Because Spatial data mining must consider the topology relation between the neighboring object, the direction and distance, etc. On the contrary, the general data mining consider every element in the database is independent. Nowadays, although the technology of the spatial data mining develops very quickly, it has not matured. There still are many things for us to do, especially, the extraction of the hedge and the feeble information from the remotely sensed images. This paper provides the background of spatial data mining, the general process of data mining, common algorithms, language and prospect.
A Study Of Spatial Data Mining Architecture And Technology

he yueshun,xu wei

DOI: https://doi.org/10.1109/ICCSIT.2009.5234974

2009-01-01

Abstract:Spatial data mining is to extract implicit knowledge, spatial relations as well as other non-explicitly information which included in the spatial database but it exist with other model from the spatial database. Due to the complexity of spatial data and its application of the professional, and it gives challenges to spatial data mining. In the paper, the characteristics of spatial datum were analyzed, and the distinctions were presented between spatial data and the traditional relationship data, the traditional data mining processes and architecture were Analyzed and improved, A new process and architecture were presented for spatial datum data mining based on data warehouse, On the basis of the work, we design a prototype system for data mining of the spatial datum, We resolved and implemented the synergetic work in ingenious between the layers of the prototype system and between the components in the layer by the four-tier of J2EE architecture, Also, an implementation method was presented for the key technology.
Research on Progress of Spatial Data Mining

HUANG Tian-qiang,WANG Jin-dong,ZHANG Lei

DOI: https://doi.org/10.3969/j.issn.1672-4143.2006.05.012

2006-01-01

Abstract:Comparison between spatial data mining and traditional data mining is presented.The main approaches of spatial data mining and characteristics of them are discussed according to classifying them by motive,especially paying attention to approaches including spatial generalization,spatial association mining,spatial classification,spatial trend detecting,spatial clustering,spatial outliers detecting.A review of advance of spatial data mining is proposed.In conclusion,the main trends of spatial data mining are forecasted.
Spatial Data Cube: Provides Better Support for Spatial Data Mining

YZ Zhang,K Xie,X Ma,D Xu,C Cai,S Tang

DOI: https://doi.org/10.1109/igarss.2005.1525227

2005-01-01

Abstract:Spatial data mining is a promising technique that deals with extraction of implicit knowledge or other interesting patterns from large amount of spatial data. Though most data mining systems work with data stored in flat files or operational database, it has been recognized that mining in a data warehouse usually result in better information. Because data are usually cleansed before they are stored into data warehouse. Furthermore, data warehouse provides data with different levels of summarization for the clients, which will lead to fruitful data mining. However, current techniques of data warehouse can not handle spatial data well. Both dimensions and measures in the data model of data warehouse are nonspatial data. In this paper, we propose a new data model called spatial data cube for data warehouse. Spatial data cube supports both spatial and nonspatial data. We also introduce how to construct a spatial data cube that can answer queries efficiently by selective materialization. We believe that the spatial data cube can provide better support for spatial data mining.
A Novel Method To Integrate Spatial Data Mining And Geographic Information System

Xx Jin,Yk Cai,Kq Xie,Xj Ma,Yx Sun,C Cai

DOI: https://doi.org/10.1109/IGARSS.2005.1525219

2005-01-01

Abstract:GIS is it good spatial analysis tool, nevertheless, as the accumulation of spatial data, the functions that GIS offers are not enough. Spatial data mining, which can automatically discover implicit knowledge from spatial data, has recently received wide attentions. However, the spatial data mining. systems are not competent for preprocessing and presenting spatial data. Hence, the idea to integrate spatial data mining and GIS is straight forward. In this paper, we propose a novel method to integrate spatial data mining and GIS. This method is implemented with SPMML, an XML-based language that extended from PMML. We build a prototype system with this method and it works well.
Spatial Data Mining with Uncertainty

Binbin He,Cuihua Chen

DOI: https://doi.org/10.1007/978-3-540-74377-4_33

2006-01-01

Abstract:On the basis of analyzing the deficiencies of traditional spatial data mining, a framework for spatial data mining with uncertainty has been founded. Four key problems have been analyzed, including uncertainty simulation of spatial data with Monte Carlo method, spatial autocorrelation measurement, discretization of continuous data based on neighbourhood EM algorithm and uncertainty assessment of association rules. Meanwhile, the experiments concerned have been performed using the environmental geochemistry data gotten from Dexing, Jiangxi province in China.
Research on the Methods of Data Reduction in Data Preprocessing

Guoqing Wang

2008-01-01

Abstract:Data selection and data preparation applied before data mining help improving the efficiency of the mining process and the quality of the mining results.Among these,data reduction techniques can be applied to obtain a reduced representation of the data set that is smaller in volume and speed up the mining process.This paper introduces some typical methods of data reduction,describes the general application problems,analyses some popular characteristics.Also,we do some experimental works on data reduction techniques through the studies of the practical data.
Preliminary Research in Using the Technology of Data Mining to Analyze Remote Sensing Data

XF Zhang,XingLi,X Zhang,QX Tong,W Liu,LF Zheng

DOI: https://doi.org/10.1117/12.539951

2003-01-01

Abstract:Remote sensing data, especially the hyperspectral remote sensing data, characterize their great quantities. So how to deal wtih these data is a focus. Database has solved the problem of storing, searching, updating and maintaining of the data, but it is not satisfactory in disposing them. In recent years, the technology of data warehouse has great development. It can re-integrate, synthesize and separate the data of database, and use the searching pattern of multiple dimensions to realize data mining (DM). This technolgoy has been widely used in commerce to analyze the inner relationship of the numerous data and makes some remarkable achievements in decision supporting. Data warehouse and Data mining technology have been used in GIS. This article would give a set of complete steps and some general methods in using the DM to analyze the remote sensing data, especially in hyperspectral data. And it tries to do some preliminary exploration in using it to deeply analyze the potential relations among the acquired spectra, images and biology parameters of the experiments and get some anticipated possible results.
Data Preprocessing: The Techniques for Preparing Clean and Quality Data for Data Analytics Process

Ashish P. Joshi,Biraj V. Patel

DOI: https://doi.org/10.13005/ojcst13.0203.03

2021-01-30

Oriental journal of computer science and technology

Abstract:The model and pattern for real time data mining have an important role for decision making. The meaningful real time data mining is basically depends on the quality of data while row or rough data available at warehouse. The data available at warehouse can be in any format, it may huge or it may unstructured. These kinds of data require some process to enhance the efficiency of data analysis. The process to make it ready to use is called data preprocessing. There can be many activities for data preprocessing such as data transformation, data cleaning, data integration, data optimization and data conversion which are use to converting the rough data to quality data. The data preprocessing techniques are the vital step for the data mining. The analyzed result will be good as far as data quality is good. This paper is about the different data preprocessing techniques which can be use for preparing the quality data for the data analysis for the available rough data.
Uncertain Spatial Data Mining Algorithms

Bin-bin HE,Tao FANG,Da-zhi GUO

DOI: https://doi.org/10.3321/j.issn:1000-1964.2007.01.024

2007-01-01

Abstract:On the basis of analyzing the uncertainties of spatial data and spatial data mining (SDM), in view of the limits of traditional spatial data mining, the uncertainties of spatial data and spatial data mining were properly combined. Based on EM algorithm and Apriori algorithm, the uncertain spatial data mining algorithms, including uncertain spatial clustering algorithm (uncertain neighborhood expectation maximum, UNEM) and uncertain spatial association rules mining (USAR) model were presented. Meanwhile, the experiments concerned have been performed using the geo-spatial data from 37 typified cites in China. The results show that the knowledge acquired through the uncertain spatial data mining algorithms is more real and objective than those through traditional spatial data mining methods.
Data Preprocessing of Classification Mining of Credit Card

翟凌慧,马少平,唐焕玲

DOI: https://doi.org/10.3969/j.issn.1000-3428.2003.11.075

2003-01-01

Abstract:This paper describes the data preprocessing of classification minin g of credit card. The database tables concerned with analysis topic from many ap plication data are selected, and the input information of training samples from these tables through a set of database operations can be got. Then, many parame ters used to measure classification standard from history transaction data are a bstracted, and training samples are classified manually. At last, the paper hand les with problems in the training data such as noisy data, missing values and no rmalization, and obtains training sample data needed by classification mining fi nally.
Spatial data mining-focused research on image processing techniques for remotely sensed images

Yongxia Fan,Shizhou Feng,Jing Du

DOI: https://doi.org/10.1080/13682199.2024.2410673

2024-10-20

The Imaging Science Journal

Abstract:Spatial data mining is an important approach for collecting useful data from big datasets, especially remotely sensed images. This study tackles issues in environmental monitoring and management using sophisticated image processing. The Horse Herd Optimization-based VGG19 (HHO-VGG19) is proposed to improve land cover classification, recognition of objects, detection of changes, and detection of anomalies. The study used the BCDD dataset, which was scaled to 512 × 512 pixels, then applied Z-score normalization and extracted features using Principal Component Analysis (PCA). The VGG19 architecture was enhanced by utilizing Horse Herd Optimization to enhance image classification efficiency. The HHO-VGG19 model surpasses conventional techniques, with F1-score of 92%, a recall of 94%, an accuracy of 98.5%, and a 30-second execution time reduction. The findings indicate the efficiency of integrating sophisticated image processing with spatial data mining, giving an effective tool for remote sensing image processing in environmental uses including tracking ecosystems and handling of natural resources.

imaging science & photographic technology
Techniques of Spatial Data Analysis

WANG Jin-feng,WU Ji-lei,SUN Ying-jun,LI Lian-fa,MENG Bin

DOI: https://doi.org/10.3321/j.issn:1000-0585.2005.03.015

2005-01-01

Geographical Research

Abstract:GIS is a geo-referenced data based system for manipulating digital spatial and thematic data. Because of its unique properties of spatial data (spatial dependence and spatial heterogeneity), there is an urgent need to investigate the mechanisms of spatial processes through exploring spatial data. The techniques of spatial analysis can be categorized into six groups: (1) generation of spatial data and its preliminary treatment; (2) spatialization of attribute data and scale transformation of spatial data; (3) visualization and exploration of spatial data; (4) geo-statistics; (5) analysis of lattice data; and (6) reversing information from complicated spatial data. To meet the real world solutions, a procedural diagram of spatial data analysis is suggested as below: (1) preliminary data treatment; (2) modeling; (3) solution; and (4) interpretation. Various techniques are being employed to investigate spatial processes, the identical study objective and some common characteristics of the study object may lead to the integration and convergence of the various approaches, and further the introduction of a general science: spatial mathematics. A same important issue is to develop robust spatial analysis computer packages for public uses.
Big Geodata Mining: Objective, Connotations and Research Issues

Tao Pei,Ci Song,Sihui Guo,Hua Shu,Yaxi Liu,Yunyan Du,Ting Ma,Chenghu Zhou

DOI: https://doi.org/10.1007/s11442-020-1726-7

2020-01-01

Journal of Geographical Sciences

Abstract:The objective, connotations and research issues of big geodata mining were discussed to address its significance to geographical research in this paper. Big geodata may be categorized into two domains: big earth observation data and big human behavior data. A description of big geodata includes, in addition to the "5Vs" (volume, velocity, value, variety and veracity), a further five features, that is, granularity, scope, density, skewness and precision. Based on this approach, the essence of mining big geodata includes four aspects. First, flow space, where flow replaces points in traditional space, will become the new presentation form for big human behavior data. Second, the objectives for mining big geodata are the spatial patterns and the spatial relationships. Third, the spatiotemporal distributions of big geodata can be viewed as overlays of multiple geographic patterns and the characteristics of the data, namely heterogeneity and homogeneity, may change with scale. Fourth, data mining can be seen as a tool for discovery of geographic patterns and the patterns revealed may be attributed to human-land relationships. The big geodata mining methods may be categorized into two types in view of the mining objective, i.e., classification mining and relationship mining. Future research will be faced by a number of issues, including the aggregation and connection of big geodata, the effective evaluation of the mining results and the challenge for mining to reveal "non-trivial" knowledge.
Evaluating Spatial Data Quality in GIS Database

Ying Su,Lei Yang,Zhanming Jin

DOI: https://doi.org/10.1109/wicom.2007.1463

2007-01-01

Abstract:The quality of spatial data is often limited by the quality of their sources such as paper maps and satellite images. Spatial operations performed on database of geographical information systems (GIS) such as selection, projection, and Cartesian product, do not always work correctly because their accuracy and completeness depends on the quality of spatial data. The present paper suggests a methodology to evaluate two data quality characteristics - accuracy and completeness - of the spatial database. Four quantitative measures are introduced to assess the quality of spatial data. Their explicit forms are derived for a tuple, and four assumptions are presented where the measures can be evaluated efficiently by numerical calculation.
Outlier Detection and Spatial Analysis Algorithms

Jacob John

DOI: https://doi.org/10.48550/arXiv.2106.10669

2021-06-20

Abstract:Outlier detection is a significant area in data mining. It can be either used to pre-process the data prior to an analysis or post the processing phase (before visualization) depending on the effectiveness of the outlier and its importance. Outlier detection extends to several fields such as detection of credit card fraud, network intrusions, machine failure prediction, potential terrorist attacks, and so on. Outliers are those data points with characteristics considerably different. They deviate from the data set causing inconsistencies, noise and anomalies during analysis and result in modification of the original points However, a common misconception is that outliers have to be immediately eliminated or replaced from the data set. Such points could be considered useful if analyzed separately as they could be obtained from a separate mechanism entirely making it important to the research question. This study surveys the different methods of outlier detection for spatial analysis. Spatial data or geospatial data are those that exhibit geographic properties or attributes such as position or areas. An example would be weather data such as precipitation, temperature, wind velocity, and so on collected for a defined region.

Machine Learning

Data Preprocessing of Spatial Data Mining

Research on Data Preprocessing Methods for Big Data

A Combined Data Preprocessing Method Based on K-means Clustering and Singular Spectrum Analysis

Data Mining In Multisensor System Based On Rough Set Theory

QUALITY ASSESSMENT AND UNCERTAINTY HANDLING IN SPATIAL DATA MINING

The Technology and Methods of Spatial Data Mining and Their Application

A Study Of Spatial Data Mining Architecture And Technology

Research on Progress of Spatial Data Mining

Spatial Data Cube: Provides Better Support for Spatial Data Mining

A Novel Method To Integrate Spatial Data Mining And Geographic Information System

Spatial Data Mining with Uncertainty

Research on the Methods of Data Reduction in Data Preprocessing

Preliminary Research in Using the Technology of Data Mining to Analyze Remote Sensing Data

Data Preprocessing: The Techniques for Preparing Clean and Quality Data for Data Analytics Process

Uncertain Spatial Data Mining Algorithms

Data Preprocessing of Classification Mining of Credit Card

Spatial data mining-focused research on image processing techniques for remotely sensed images

Techniques of Spatial Data Analysis

Big Geodata Mining: Objective, Connotations and Research Issues

Evaluating Spatial Data Quality in GIS Database

Outlier Detection and Spatial Analysis Algorithms