Query Processing over Incomplete Databases

Yunjun Gao,Xiaoye Miao

DOI: https://doi.org/10.1007/978-3-031-01863-3

2018-01-01

Synthesis Lectures on Data Management

Abstract:Incomplete data is part of life and almost all areas of scientific studies. Users tend to skip certain fields when they fill out online forms; participants choose to ignore sensitive questions on surv

What problem does this paper attempt to address?

Incomplete Data Management: a Survey.

Xiaoye Miao,Yunjun Gao,Su Guo,Wanqi Liu

DOI: https://doi.org/10.1007/s11704-016-6195-x

IF: 2.6688

2017-01-01

Frontiers of Computer Science

Abstract:Incomplete data accompanies our life processes and covers almost all fields of scientific studies, as a result of delivery failure, no power of battery, accidental loss, etc. However, how to model, index, and query incomplete data incurs big challenges. For example, the queries struggling with incomplete data usually have dissatisfying query results due to the improper incompleteness handling methods. In this paper, we systematically review the management of incomplete data, including modelling, indexing, querying, and handling methods in terms of incomplete data. We also overview several application scenarios of incomplete data, and summarize the existing systems related to incomplete data. It is our hope that this survey could provide insights to the database community on how incomplete data is managed, and inspire database researchers to develop more advanced processing techniques and tools to cope with the issues resulting from incomplete data in the real world.
Processing Methods for Incomplete Information Systems Based on Rough Sets

Ren-pu LI,Dao HUANG

DOI: https://doi.org/10.3969/j.issn.1006-3080.2005.02.023

2005-01-01

Abstract:How to deal with missing values of information system is an important part of data mining. Recent methods based on rough set for incomplete information system are introduced and analyzed from three aspects, extension of rough set theory, attribute reduction and rule extraction. Future research (directions) are also proposed. This paper provides the basis for seeking efficient mining algorithm for (incomplete) information systems.
Query Processing on Uncertain Data

JIANG Tao,GAO Yun-jun,ZHANG Bin,ZHOU Ao-ying,YUE Guang-xue

DOI: https://doi.org/10.3969/j.issn.2095-6649.2013.05.012

2013-01-01

Abstract:Data uncertainty is pervasive in various fields,for example,economy,military,logistic,finance and telecommunication,etc.Uncertain data are inherent in some important applications,such as environmental surveillance,market analysis,Location-Based Service(LBS),and quantitative economics research.Due to the importance of those applications and the rapidly increasing amount of uncertain data collected and accumulated,querying large collections of uncertain data has become an important task and has received more and more attention from the database community in recent years.This paper introduces the principle of uncertain data query,and surveys the advance of the research on uncertain data query processing,including Nearest Neighbor(NN)query,Reverse Nearest Neighbor(RNN)query,Ranking query,top-k query and join query.By a detailed comparison,the pros and cons of the techniques are discussed.In the end,the problems in current research and some future research issues are outlined.
A Survey on Incomplete Multi-view Clustering

Jie Wen,Zheng Zhang,Lunke Fei,Bob Zhang,Yong Xu,Zhao Zhang,Jinxing Li

DOI: https://doi.org/10.1109/TSMC.2022.3192635

2022-08-17

Abstract:Conventional multi-view clustering seeks to partition data into respective groups based on the assumption that all views are fully observed. However, in practical applications, such as disease diagnosis, multimedia analysis, and recommendation system, it is common to observe that not all views of samples are available in many cases, which leads to the failure of the conventional multi-view clustering methods. Clustering on such incomplete multi-view data is referred to as incomplete multi-view clustering. In view of the promising application prospects, the research of incomplete multi-view clustering has noticeable advances in recent years. However, there is no survey to summarize the current progresses and point out the future research directions. To this end, we review the recent studies of incomplete multi-view clustering. Importantly, we provide some frameworks to unify the corresponding incomplete multi-view clustering methods, and make an in-depth comparative analysis for some representative methods from theoretical and experimental perspectives. Finally, some open problems in the incomplete multi-view clustering field are offered for researchers.

Machine Learning,Multimedia
Searching Dimension Incomplete Databases

Wei Cheng,Xiaoming Jin,Jian-Tao Sun,Xuemin Lin,Xiang Zhang,Wei Wang

DOI: https://doi.org/10.1109/tkde.2013.14

IF: 9.235

2013-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Similarity query is a fundamental problem in database, data mining and information retrieval research. Recently, querying incomplete data has attracted extensive attention as it poses new challenges to traditional querying techniques. The existing work on querying incomplete data addresses the problem where the data values on certain dimensions are unknown. However, in many real-life applications, such as data collected by a sensor network in a noisy environment, not only the data values but also the dimension information may be missing. In this work, we propose to investigate the problem of similarity search on dimension incomplete data. A probabilistic framework is developed to model this problem so that the users can find objects in the database that are similar to the query with probability guarantee. Missing dimension information poses great computational challenge, since all possible combinations of missing dimensions need to be examined when evaluating the similarity between the query and the data objects. We develop the lower and upper bounds of the probability that a data object is similar to the query. These bounds enable efficient filtering of irrelevant data objects without explicitly examining all missing dimension combinations. A probability triangle inequality is also employed to further prune the search space and speed up the query process. The proposed probabilistic framework and techniques can be applied to both whole and subsequence queries. Extensive experimental results on real-life data sets demonstrate the effectiveness and efficiency of our approach.
Querying Incomplete Numerical Data: Between Certain and Possible Answers

Marco Console,Leonid Libkin,Liat Peterfreund

DOI: https://doi.org/10.48550/arXiv.2210.15395

2022-11-01

Abstract:Queries with aggregation and arithmetic operations, as well as incomplete data, are common in real-world database, but we lack a good understanding of how they should interact. On the one hand, systems based on SQL provide ad-hoc rules for numerical nulls, on the other, theoretical research largely concentrates on the standard notions of certain and possible answers. In the presence of numerical attributes and aggregates, however, these answers are often meaningless, returning either too little or too much. Our goal is to define a principled framework for databases with numerical nulls and answering queries with arithmetic and aggregations over them. Towards this goal, we assume that missing values in numerical attributes are given by probability distributions associated with marked nulls. This yields a model of probabilistic bag databases in which tuples are not necessarily independent, since nulls can repeat. We provide a general compositional framework for query answering, and then concentrate on queries that resemble standard SQL with arithmetic and aggregation. We show that these queries are measurable, and that their outputs have a finite representation. Moreover, since the classical forms of answers provide little information in the numerical setting, we look at the probability that numerical values in output tuples belong to specific intervals. Even though their exact computation is intractable, we show efficient approximation algorithms to compute such probabilities.

Databases
Processing Incomplete K Nearest Neighbor Search

Xiaoye Miao,Yunjun Gao,Gang Chen,Baihua Zheng,Huiyong Cui

DOI: https://doi.org/10.1109/tfuzz.2016.2516562

IF: 12.253

2016-01-01

IEEE Transactions on Fuzzy Systems

Abstract:Given a setS of multidimensional objects and a query object q, a k nearest neighbor (kNN) query finds from S the k closest objects to q. This query is a fundamental problem in database, data mining, and information retrieval research. It plays an important role in a wide spectrum of real applications such as image recognition and location-based services. However, due to the failure of data transmission devices, improper storage, and accidental loss, incomplete data exist widely in those applications, where some dimensional values of data items are missing. In this paper, we systematically study incomplete k nearest neighbor (IkNN) search, which aims at the kNN query for incomplete data. We formalize this problem and propose an efficient lattice partition algorithm using our newly developed LαB index to support exact IkNN retrieval, with the help of two pruning heuristics, i.e., α value pruning and partial distance pruning. Furthermore, we propose an approximate algorithm, namely histogram approximate, to support approximate IkNN search with improved search efficiency and guaranteed error bound. Extensive experiments using both real and synthetic datasets demonstrate the effectiveness of newly designed indexes and pruning heuristics, as well as the performance of our presented algorithms under a variety of experimental settings.
On Efficient k-Skyband Query Processing over Incomplete Data.

Xiaoye Miao,Yunjun Gao,Lu Chen,Gang Chen,Qing Li,Tao Jiang

DOI: https://doi.org/10.1007/978-3-642-37487-6_32

2013-01-01

Abstract:The Skyline query and its variants have been extensively explored in the literature. Existing approaches, except one, assume that all dimensions are available for all data items. However, many practical applications such as sensor networks, decision making, and location-based services, may involve incomplete data items, i.e., some dimensional values are missing , due to the device failure or the privacy preservation. In this paper, for the first time, we study the problem of efficient k-Skyband (kSB) query processing on incomplete data, where multi-dimensional data items are missing some values of their dimensions. We formalize the problem, and then present several efficient algorithms for tackling it. Our methods employ some novel concepts/structures (e.g., expired skyline, shadow skyline, thickness warehouse, etc.) to improve the search performance. Extensive experiments with both real and synthetic data sets demonstrate the effectiveness and efficiency of our proposed algorithms. © Springer-Verlag 2013.
Probabilistic Similarity Query on Dimension Incomplete Data

Wei Cheng,Xiaoming Jin,Jian-Tao Sun

DOI: https://doi.org/10.1109/ICDM.2009.72

2009-01-01

Abstract:Retrieving similar data has drawn many research efforts in the literature due to its importance in data mining, database and information retrieval. This problem is challenging when the data is incomplete. In previous research, data incompleteness refers to the fact that data values for some dimensions are unknown. However, in many practical applications (e.g., data collection by sensor network under bad environment), not only data values but even data dimension information may also be missing, which will make most similarity query algorithms infeasible. In this work, we propose the novel similarity query problem on dimension incomplete data and adopt a probabilistic framework to model this problem. For this problem, users can give a distance threshold and a probability threshold to specify their retrieval requirements. The distance threshold is used to specify the allowed distance between query and data objects and the probability threshold is used to require that the retrieval results satisfy the distance condition at least with the given probability. Instead of enumerating all possible cases to recover the missed dimensions, we propose an efficient approach to speed up the retrieval process by leveraging the inherent relations between query and dimension incomplete data objects. During the query process, we estimate the lower/upper bounds of the probability that the query is satisfied by a given data object, and utilize these bounds to filter irrelevant data objects efficiently. Furthermore, a probability triangle inequality is proposed to further speed up query processing. According to our experiments on real data sets, the proposed similarity query method is verified to be effective and efficient on dimension incomplete data.
Processing K-Skyband, Constrained Skyline, and Group-by Skyline Queries on Incomplete Data

Yunjun Gao,Xiaoye Miao,Huiyong Cui,Gang Chen,Qing Li

DOI: https://doi.org/10.1016/j.eswa.2014.02.033

IF: 8.5

2014-01-01

Expert Systems with Applications

Abstract:The skyline operator has been extensively explored in the literature, and most of the existing approaches assume that all dimensions are available for all data items. However, many practical applications such as sensor networks, decision making, and location-based services, may involve incomplete data items, i.e., some dimensional values are missing, due to the device failure or the privacy preservation. This paper is the first, to our knowledge, study of k-skyband (kSB) query processing on incomplete data, where multi-dimensional data items are missing some values of their dimensions. We formalize the problem, and then present two efficient algorithms for processing it. Our methods introduce some novel concepts including expired skyline, shadow skyline, and thickness warehouse, in order to boost the search performance. As a second step, we extend our techniques to tackle constrained skyline (CS) and group-by skyline (GBS) queries over incomplete data. Extensive experiments with both real and synthetic data sets demonstrate the effectiveness and efficiency of our proposed algorithms under various experimental settings. (C) 2014 Elsevier Ltd. All rights reserved.
SQL Query Completion for Data Exploration

Marie Le Guilly,Jean-Marc Petit,Vasile-Marian Scuturici

DOI: https://doi.org/10.48550/arXiv.1802.02872

2018-02-08

Abstract:Within the big data tsunami, relational databases and SQL are still there and remain mandatory in most of cases for accessing data. On the one hand, SQL is easy-to-use by non specialists and allows to identify pertinent initial data at the very beginning of the data exploration process. On the other hand, it is not always so easy to formulate SQL queries: nowadays, it is more and more frequent to have several databases available for one application domain, some of them with hundreds of tables and/or attributes. Identifying the pertinent conditions to select the desired data, or even identifying relevant attributes is far from trivial. To make it easier to write SQL queries, we propose the notion of SQL query completion: given a query, it suggests additional conditions to be added to its WHERE clause. This completion is semantic, as it relies on the data from the database, unlike current completion tools that are mostly syntactic. Since the process can be repeated over and over again -- until the data analyst reaches her data of interest --, SQL query completion facilitates the exploration of databases. SQL query completion has been implemented in a SQL editor on top of a database management system. For the evaluation, two questions need to be studied: first, does the completion speed up the writing of SQL queries? Second , is the completion easily adopted by users? A thorough experiment has been conducted on a group of 70 computer science students divided in two groups (one with the completion and the other one without) to answer those questions. The results are positive and very promising.

Databases
Survey on Query Processing Based on Uncertain Data

崔斌,卢阳

DOI: https://doi.org/10.3724/sp.j.1087.2008.02729

2009-01-01

Abstract:Uncertain data is inherent in some important application fields,such as sensor networks and mobile object tracking.Using traditional querying methods on uncertain data will bias the answer set,and hence cannot satisfy users' needs.Therefore,query processing based on uncertain data has attracted more and more attention.Different from queries on certain data,research work on uncertain data introduce probability into data model to measure the likeness of an uncertain object as one element of the answer set.Due to different problem definitions and data models,query types differentiate from each other greatly.This survey introduced range queries,top-k queries and skyline queries based on uncertain data from the views of problem definitions,data models,pruning strategies and algorithms.
Approximate Query Processing: What is New and Where to Go?: A Survey on Approximate Query Processing

Kaiyu Li,Guoliang Li

DOI: https://doi.org/10.1007/s41019-018-0074-4

2018-01-01

Data Science and Engineering

Abstract:Online analytical processing (OLAP) is a core functionality in database systems. The performance of OLAP is crucial to make online decisions in many applications. However, it is rather costly to support OLAP on large datasets, especially big data, and the methods that compute exact answers cannot meet the high-performance requirement. To alleviate this problem, approximate query processing (AQP) has been proposed, which aims to find an approximate answer as close as to the exact answer efficiently. Existing AQP techniques can be broadly categorized into two categories. (1) Online aggregation: select samples online and use these samples to answer OLAP queries. (2) Offline synopses generation: generate synopses offline based on a-priori knowledge (e.g., data statistics or query workload) and use these synopses to answer OLAP queries. We discuss the research challenges in AQP and summarize existing techniques to address these challenges. In addition, we review how to use AQP to support other complex data types, e.g., spatial data and trajectory data, and support other applications, e.g., data visualization and data cleaning. We also introduce existing AQP systems and summarize their advantages and limitations. Lastly, we provide research challenges and opportunities of AQP. We believe that the survey can help the partitioners to understand existing AQP techniques and select appropriate methods in their applications.
Survey of Database Usability for Query Results

Liu Qing,Gao Yunjun

DOI: https://doi.org/10.7544/issn1000-1239.2017.20160806

2017-01-01

Journal of Computer Research and Development

Abstract:Database usability has received much attention in the database community because of its importance.The goal of database usability is to help users utilize database more efficiently and conveniently,and thus improving the user's satisfaction for the database.In this survey,we focus on the database usability for query results.Currently,the queries only return the query results to users.If the query result is unexpected for the users,it will frustrate users.However,the database system neither gives explanations for the unexpected query results,nor offers any suggestion on how to get the expected results for users.The users only can debug the queries by themselves,which is cumbersome and time-consuming.If the database system can offer such explanations and suggestions,it helps the users understand initial query better,and know how to change the query until the satisfactory results are found,hence improving the usability of the database.Towards this,the studies on unexpected query results have been explored.In this paper,we provide a comprehensive survey of the most recent research on database usability for query results.The paper first analyses the unexpected query results,and introduces the corresponding three problems,i.e.,causality & responsibility,why-not & why questions,and why-few & why-many questions,and highlights the importance of these three problems.Then,the state of the art progresses of the unexpected query result research have been surveyed and summarized.Finally,the paper raises some directions for the future work.
Top-k Dominating Queries on Incomplete Data

Xiaoye Miao,Yunjun Gao,Baihua Zheng,Gang Chen,Huiyong Cui

DOI: https://doi.org/10.1109/tkde.2015.2460742

2016-01-01

Abstract:The top-k dominating (TKD) query returns the k objects that dominate the maximum number of objects in a given dataset. It combines the advantages of skyline and top-k queries, and plays an important role in many decision support applications. Incomplete data exists in a wide spectrum of real datasets, due to device failure, privacy preservation, data loss, and so on. In this paper, for the first time, we carry out a systematic study of TKD queries on incomplete data, which involves the data having some missing dimensional value(s). We formalize this problem, and propose a suite of efficient algorithms for answering TKD queries over incomplete data. Our methods employ some noveltechniques, such as upper bound score pruning, bitmap pruning, and partial score pruning, to boost query efficiency. Extensive experimental evaluation using both real and synthetic datasets demonstrates the effectiveness of our developed pruning heuristics and the performance of our presented algorithms.
Contextual Ranking of Query Results with Incomplete Preferences

Xiang Li,Ling Feng,Lizhu Zhou

DOI: https://doi.org/10.1109/jcpc.2009.5420161

2009-01-01

Abstract:Context-aware database is frequently used in user-centric applications. The users normally tend to express their preferences through comparisons. For example, Bob prefers Cornell to USC. Because the users only compare a small proportion of the products under difference contexts, "incomplete" preferences are a common occurrence. We propose a ranking approach which can contextually rank the query results when the users preferences are incomplete. We empirically evaluate the practical effectiveness of our method.
Approximate Query Processing: What is New and Where to Go?

Kaiyu Li,Guoliang Li

DOI: https://doi.org/10.1007/s41019-018-0074-4

2018-09-14

Data Science and Engineering

Abstract:Online analytical processing (OLAP) is a core functionality in database systems. The performance of OLAP is crucial to make online decisions in many applications. However, it is rather costly to support OLAP on large datasets, especially big data, and the methods that compute exact answers cannot meet the high-performance requirement. To alleviate this problem, approximate query processing (AQP) has been proposed, which aims to find an approximate answer as close as to the exact answer efficiently. Existing AQP techniques can be broadly categorized into two categories. (1) Online aggregation: select samples online and use these samples to answer OLAP queries. (2) Offline synopses generation: generate synopses offline based on a-priori knowledge (e.g., data statistics or query workload) and use these synopses to answer OLAP queries. We discuss the research challenges in AQP and summarize existing techniques to address these challenges. In addition, we review how to use AQP to support other complex data types, e.g., spatial data and trajectory data, and support other applications, e.g., data visualization and data cleaning. We also introduce existing AQP systems and summarize their advantages and limitations. Lastly, we provide research challenges and opportunities of AQP. We believe that the survey can help the partitioners to understand existing AQP techniques and select appropriate methods in their applications.
AQP++: Connecting Approximate Query Processing with Aggregate Precomputation for Interactive Analytics

Jinglin Peng,Dongxiang Zhang,Jiannan Wang,Jian Pei

DOI: https://doi.org/10.1145/3183713.3183747

2018-01-01

Abstract:Interactive analytics requires database systems to be able to answer aggregation queries within interactive response times. As the amount of data is continuously growing at an unprecedented rate, this is becoming increasingly challenging. In the past, the database community has proposed two separate ideas, sampling-based approximate query processing (AQP) and aggregate precomputation (AggPre) such as data cubes, to address this challenge. In this paper, we argue for the need to connect these two separate ideas for interactive analytics. We propose AQP++, a novel framework to enable the connection. The framework can leverage both a sample as well as a precomputed aggregate to answer user queries. We discuss the advantages of having such a unified framework and identify new challenges to fulfill this vision. We conduct an in-depth study of these challenges for range queries and explore both optimal and heuristic solutions to address them. Our experiments using two public benchmarks and one real-world dataset show that AQP++ achieves a more flexible and better trade-off among preprocessing cost, query response time, and answer quality than AQP or AggPre.
Possible and Certain Answers for Queries over Order-Incomplete Data

Antoine Amarilli,Mouhamadou Lamine Ba,Daniel Deutch,Pierre Senellart

DOI: https://doi.org/10.4230/LIPIcs.TIME.2017.4

2018-01-27

Abstract:To combine and query ordered data from multiple sources, one needs to handle uncertainty about the possible orderings. Examples of such "order-incomplete" data include integrated event sequences such as log entries, lists of properties (e.g., hotels and restaurants) ranked by an unknown function reflecting relevance or customer ratings, and documents edited concurrently with an uncertain order on edits. This paper introduces a query language for order-incomplete data, based on the positive relational algebra with order-aware accumulation. We use partial orders to represent order-incomplete data, and study possible and certain answers for queries in this context. We show that these problems are respectively NP-complete and coNP-complete, but identify many tractable cases depending on the query operators or input partial orders.

Databases
Querying Big Data from a Database Perspective

Wenfeng Zhao,Guohua Liu,Zhao Chen,Douglas Nyabuga,Huichun Yang,Heng Zhang,Mengfei Ni

DOI: https://doi.org/10.1109/icsai.2017.8248511

2017-01-01

Abstract:Querying big data is the cornerstone of the application of big data. From a database perspective, query is the function defined by domain, range and a specified semantics. For querying big data, the domain is the big data and becomes quite complicated with characteristics of large volume, heterogeneous types, strong timeliness, weak authenticity, etc. To describe and analyze querying big data theoretically, we propose the definitions of big data and big data system, which includes querying big data. Aforementioned characteristics of big data lead to a break-through in normal form qualification and Closed World Assumption (CWA) related to traditional database. Therefore, this paper also points out several challenges according to those characteristics of domain and analyzes them in detail using the first-order language. Since the complexity of domain makes traditionally tractable queries infeasible, this paper analyzes and summarizes the classification of queries on relational big data according to their structure and computational complexity.

Query Processing over Incomplete Databases

Incomplete Data Management: a Survey.

Processing Methods for Incomplete Information Systems Based on Rough Sets

Query Processing on Uncertain Data

A Survey on Incomplete Multi-view Clustering

Searching Dimension Incomplete Databases

Querying Incomplete Numerical Data: Between Certain and Possible Answers

Processing Incomplete K Nearest Neighbor Search

On Efficient k-Skyband Query Processing over Incomplete Data.

Probabilistic Similarity Query on Dimension Incomplete Data

Processing K-Skyband, Constrained Skyline, and Group-by Skyline Queries on Incomplete Data

SQL Query Completion for Data Exploration

Survey on Query Processing Based on Uncertain Data

Approximate Query Processing: What is New and Where to Go?: A Survey on Approximate Query Processing

Survey of Database Usability for Query Results

Top-k Dominating Queries on Incomplete Data

Contextual Ranking of Query Results with Incomplete Preferences

Approximate Query Processing: What is New and Where to Go?

AQP++: Connecting Approximate Query Processing with Aggregate Precomputation for Interactive Analytics

Possible and Certain Answers for Queries over Order-Incomplete Data

Querying Big Data from a Database Perspective