Using Additional Indexes for Fast Full-Text Search of Phrases That Contain Frequently Used Words

A. B. Veretennikov

DOI: https://doi.org/10.48550/arXiv.1801.09079

2018-11-25

Abstract:Searches for phrases and word sets in large text arrays by means of additional indexes are considered. Their use may reduce the query-processing time by an order of magnitude in comparison with standard inverted files.

Information Retrieval,Computation and Language

What problem does this paper attempt to address?

About a structure of easily updatable full-text indexes

Alexander B. Veretennikov

DOI: https://doi.org/10.48550/arXiv.2007.09377

2020-07-18

Information Retrieval

Abstract:We consider strategies to organize easily updatable associative arrays in external memory. These arrays are used for full-text search. We study indexes with different keys: single word form, two word forms, and sequences of word forms. The storage structure depends on the size of the key's data. The results of the experiments are given in the context of the proximity full-text search, which is performed by means of additional indexes.
Proximity Full-Text Search by Means of Additional Indexes with Multi-component Keys: In Pursuit of Optimal Performance

Alexander B. Veretennikov

DOI: https://doi.org/10.1007/978-3-030-23584-0_7

2019-07-10

Abstract:Full-text search engines are important tools for information retrieval. In a proximity full-text search, a document is relevant if it contains query terms near each other, especially if the query terms are frequently occurring words. For each word in a text, we use additional indexes to store information about nearby words that are at distances from the given word of less than or equal to the MaxDistance parameter. We showed that additional indexes with three-component keys can be used to improve the average query execution time by up to 94.7 times if the queries consist of high-frequency occurring words. In this paper, we present a new search algorithm with even more performance gains. We consider several strategies for selecting multi-component key indexes for a specific query and compare these strategies with the optimal strategy. We also present the results of search experiments, which show that three-component key indexes enable much faster searches in comparison with two-component key indexes. This is a pre-print of a contribution "Veretennikov A.B. (2019) Proximity Full-Text Search by Means of Additional Indexes with Multi-component Keys: In Pursuit of Optimal Performance." published in "Manolopoulos Y., Stupnikov S. (eds) Data Analytics and Management in Data Intensive Domains. DAMDID/RCDL 2018. Communications in Computer and Information Science, vol 1003" published by Springer, Cham. This book constitutes the refereed proceedings of the 20th International Conference on Data Analytics and Management in Data Intensive Domains, DAMDID/RCDL 2018, held in Moscow, Russia, in October 2018. The 9 revised full papers presented together with three invited papers were carefully reviewed and selected from 54 submissions. The final authenticated version is available online at <a class="link-external link-https" href="https://doi.org/10.1007/978-3-030-23584-0_7" rel="external noopener nofollow">this https URL</a>.

Information Retrieval
Proximity Full-Text Search with a Response Time Guarantee by Means of Additional Indexes

Alexander B. Veretennikov

DOI: https://doi.org/10.1007/978-3-030-01054-6_66

2018-11-19

Abstract:Full-text search engines are important tools for information retrieval. Term proximity is an important factor in relevance score measurement. In a proximity full-text search, we assume that a relevant document contains query terms near each other, especially if the query terms are frequently occurring words. A methodology for high-performance full-text query execution is discussed. We build additional indexes to achieve better efficiency. For a word that occurs in the text, we include in the indexes some information about nearby words. What types of additional indexes do we use? How do we use them? These questions are discussed in this work. We present the results of experiments showing that the average time of search query execution is 44-45 times less than that required when using ordinary inverted indexes. This is a pre-print of a contribution "Veretennikov A.B. Proximity Full-Text Search with a Response Time Guarantee by Means of Additional Indexes" published in "Arai K., Kapoor S., Bhatia R. (eds) Intelligent Systems and Applications. IntelliSys 2018. Advances in Intelligent Systems and Computing, vol 868" published by Springer, Cham. The final authenticated version is available online at: <a class="link-external link-https" href="https://doi.org/10.1007/978-3-030-01054-6_66" rel="external noopener nofollow">this https URL</a>. The work was supported by Act 211 Government of the Russian Federation, contract no 02.A03.21.0006.

Information Retrieval
Relevance ranking for proximity full-text search based on additional indexes with multi-component keys

Alexander B. Veretennikov

DOI: https://doi.org/10.35634/vm210110

2021-08-01

Abstract:The problem of proximity full-text search is considered. If a search query contains high-frequently occurring words, then multi-component key indexes deliver an improvement in the search speed compared with ordinary inverted indexes. It was shown that we can increase the search speed by up to 130 times in cases when queries consist of high-frequently occurring words. In this paper, we investigate how the multi-component key index architecture affects the quality of the search. We consider several well-known methods of relevance ranking, where these methods are of different authors. Using these methods, we perform the search in the ordinary inverted index and then in an index enhanced with multi-component key indexes. The results show that with multi-component key indexes we obtain search results that are very close, in terms of relevance ranking, to the search results that are obtained by means of ordinary inverted indexes.

Information Retrieval
Proximity full-text searches of frequently occurring words with a response time guarantee

Alexander B. Veretennikov

DOI: https://doi.org/10.1007/978-3-030-42176-2_37

2020-09-06

Abstract:Full-text search engines are important tools for information retrieval. In a proximity full-text search, a document is relevant if it contains query terms near each other, especially if the query terms are frequently occurring words. For each word in the text, we use additional indexes to store information about nearby words at distances from the given word of less than or equal to MaxDistance, which is a parameter. A search algorithm for the case when the query consists of high-frequently used words is discussed. In addition, we present results of experiments with different values of MaxDistance to evaluate the search speed dependence on the value of MaxDistance. These results show that the average time of the query execution with our indexes is 94.7-45.9 times (depending on the value of MaxDistance) less than that with standard inverted files when queries that contain high-frequently occurring words are evaluated. This is a pre-print of a contribution published in Pinelas S., Kim A., Vlasov V. (eds) Mathematical Analysis With Applications. CONCORD-90 2018. Springer Proceedings in Mathematics & Statistics, vol 318, published by Springer, Cham. The final authenticated version is available online at: <a class="link-external link-https" href="https://doi.org/10.1007/978-3-030-42176-2_37" rel="external noopener nofollow">this https URL</a>

Information Retrieval
Efficient Phrase Querying with Flat Position Index

Dongdong Shan,Wayne Xin Zhao,Jing He,Rui Yan,Hongfei Yan,Xiaoming Li

DOI: https://doi.org/10.1145/2063576.2063875

2011-01-01

Abstract:A large proportion of search engine queries contain phrases,namely a sequence of adjacent words. In this paper, we propose to use flat position index (a.k.a schema-independent index) for phrase query evaluation. In the flat position index, the entire document collection is viewed as a huge sequence of tokens. Each token is represented by one flat position, which is a unique position offset from the beginning of the collection. Each indexed term is associated with a list of the flat positions about that term in the sequence. To recover DocID from flat positions efficiently, we propose a novel cache sensitive look-up table (CSLT), which is much faster than existing search algorithms. Experiments on TREC GOV2 data collection show that flat position index can reduce the index size and speed up phrase querying substantially, compared with traditional word-level index.
Selection of Optimal Parameters in the Fast K-Word Proximity Search Based on Multi-component Key Indexes

Alexander B. Veretennikov

DOI: https://doi.org/10.48550/arXiv.2101.03327

2021-01-09

Abstract:Proximity full-text search is commonly implemented in contemporary full-text search systems. Let us assume that the search query is a list of words. It is natural to consider a document as relevant if the queried words are near each other in the document. The proximity factor is even more significant for the case where the query consists of frequently occurring words. Proximity full-text search requires the storage of information for every occurrence in documents of every word that the user can search. For every occurrence of every word in a document, we employ additional indexes to store information about nearby words, that is, the words that occur in the document at distances from the given word of less than or equal to the MaxDistance parameter. We showed in previous works that these indexes can be used to improve the average query execution time by up to 130 times for queries that consist of words occurring with high-frequency. In this paper, we consider how both the search performance and the search quality depend on the value of MaxDistance and other parameters. Well-known GOV2 text collection is used in the experiments for reproducibility of the results. We propose a new index schema after the analysis of the results of the experiments. This is a pre-print of a contribution published in Supplementary Proceedings of the XXII International Conference on Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL 2020), Voronezh, Russia, October 13-16, 2020, P. 336-350, published by CEUR Workshop Proceedings. The final authenticated version is available online at: <a class="link-external link-http" href="http://ceur-ws.org/Vol-2790/" rel="external noopener nofollow">this http URL</a>

Information Retrieval
Two simple full-text indexes based on the suffix array

Szymon Grabowski,Marcin Raniszewski

DOI: https://doi.org/10.48550/arXiv.1405.5919

2016-05-24

Abstract:We propose two suffix array inspired full-text indexes. One, called SA-hash, augments the suffix array with a hash table to speed up pattern searches due to significantly narrowed search interval before the binary search phase. The other, called FBCSA, is a compact data structure, similar to M{ä}kinen's compact suffix array, but working on fixed sized blocks. Experiments on the Pizza~\&~Chili 200\,MB datasets show that SA-hash is about 2--3 times faster in pattern searches (counts) than the standard suffix array, for the price of requiring $0.2n-1.1n$ bytes of extra space, where $n$ is the text length, and setting a minimum pattern length. FBCSA is relatively fast in single cell accesses (a few times faster than related indexes at about the same or better compression), but not competitive if many consecutive cells are to be extracted. Still, for the task of extracting, e.g., 10 successive cells its time-space relation remains attractive.

Data Structures and Algorithms
An efficient algorithm for three-component key index construction

Alexander B. Veretennikov

DOI: https://doi.org/10.20537/vm190111

2020-06-15

Abstract:In this paper, proximity full-text searches in large text arrays are considered. A search query consists of several words. The search result is a list of documents containing these words. In a modern search system, documents that contain search query words that are near each other are more relevant than documents that do not share this trait. To solve this task, for each word in each indexed document, we need to store a record in the index. In this case, the query search time is proportional to the number of occurrences of the queried words in the indexed documents. Consequently, it is common for search systems to evaluate queries that contain frequently occurring words much more slowly than queries that contain less frequently occurring, ordinary words. For each word in the text, we use additional indexes to store information about nearby words at distances from the given word of less than or equal to MaxDistance, which is a parameter. This parameter can take a value of 5, 7, or even more. Three-component key indexes can be created for faster query execution. Previously, we presented the results of experiments showing that when queries contain very frequently occurring words, the average time of the query execution with three-component key indexes is 94.7 times less than that required when using ordinary inverted indexes. In the current work, we describe a new three-component key index building algorithm and demonstrate the correctness of the algorithm. We present the results of experiments creating such an index that is dependent on the value of MaxDistance.

Information Retrieval
Can phrase indexing help to process non-phrase queries?

Mingjie Zhu,Shuming Shi,Nenghai Yu,Ji-Rong Wen

DOI: https://doi.org/10.1145/1458082.1458174

2008-01-01

Abstract:Modern web search engines, while indexing billions of web pages, are expected to process queries and return results in a very short time. Many approaches have been proposed for efficiently computing top-k query results, but most of them ignore one key factor in the ranking functions of commercial search engines - term-proximity, which is the metric of the distance between query terms in a document. When term-proximity is included in ranking functions, most of the existing top-k algorithms will become inefficient. To address this problem, in this paper we propose to build a compact phrase index to speed up the search process when incorporating the term-proximity factor. The compact phrase index can help more accurately estimate the score upper bounds of unknown documents. The size of the phrase index is controlled by including a small portion of phrases which are possibly helpful for improving search performance. Phrase index has been used to process phrase queries in existing work. It is, however, to the best of our knowledge, the first time that phrase index is used to improve the performance of generic queries. Experimental results show that, compared with the state-of-the-art top-k computation approaches, our approach can reduce average query processing time to 1/5 for typical setttings.
Augmented Keyword Search on Spatial Entity Databases

Dongxiang Zhang,Yuchen Li,Xin Cao,Jie Shao,Heng Tao Shen

DOI: https://doi.org/10.1007/s00778-018-0497-6

2018-01-01

The VLDB Journal

Abstract:In this paper, we propose a new type of query that augments the spatial keyword search with an additional boolean expression constraint. The query is issued against a corpus of structured or semi-structured spatial entities and is very useful in applications like mobile search and targeted location-aware advertising. We devise three types of indexing and filtering strategies. First, we utilize the hybrid $\hbox {IR}^2$-tree and propose a novel hashing scheme for efficient pruning. Second, we propose an inverted index-based solution, named BE-Inv, that is more cache concious and exhibits great pruning power for boolean expression matching. Our third method, named SKB-Inv, adopts a novel two-level partitioning scheme to organize the spatial entities into inverted lists and effectively facilitate the pruning in the spatial, textual, and boolean expression dimensions. In addition, we propose an adaptive query processing strategy that takes into account the selectivity of query keywords and predicates for early termination. We conduct our experiments using two real datasets with 3.5 million Foursquare venues and 50 million Twitter geo-profiles. The results show that the methods based on inverted index are superior to the hybrid $\hbox {IR}^2$-tree; and SKB-Inv achieves the best performance.
Processing Long Queries Against Short Text

Dongxiang Zhang,Yuchen Li,Ju Fan,Lianli Gao,Fumin Shen,Heng Tao Shen

DOI: https://doi.org/10.1145/3052772

2017-01-01

Abstract:Many real applications in real-time news stream advertising call for efficient processing of long queries against short text. In such applications, dynamic news feeds are regarded as queries to match against an advertisement (ad) database for retrieving the k most relevant ads. The existing approaches to keyword retrieval cannot work well in this search scenario when queries are triggered at a very high frequency. To address the problem, we introduce new techniques to significantly improve search performance. First, we devise a two-level partitioning for tight upper bound estimation and a lazy evaluation scheme to delay full evaluation of unpromising candidates, which can bring three to four times performance boosting in a database with 7 million ads. Second, we propose a novel rank-aware block-oriented inverted index to further improve performance. In this index scheme, each entry in an inverted list is assigned a rank according to its importance in the ad. Then, we introduce a block-at-a-time search strategy based on the index scheme to support a much tighter upper bound estimation and a very early termination. We have conducted experiments with real datasets, and the results show that the rank-aware method can further improve performance by an order of magnitude.
An Efficient Approach for Building Compressed Full-Text Index for Structured Data

Jun Liang,Lin Xiao,Di Zhang

DOI: https://doi.org/10.1109/iccit.2009.42

2009-01-01

Abstract:The self-index is a kind of highly compressed, self-contained full-text index. It is designed for indexing plain texts in order to reduce its permanent storage, as well as to enhance searching performance. Apart from being a sequence of characters, usually the text has specific internal structure. The data record, as a basic model of structured data, is therefore employed to represent and organize such form of data widespread. In this paper, we design and implement an approach to building the self-index for data records via text medium. Our approach indexes the data records through an intermediate text which accommodates aligned record fields by stuffing delimiters among them. By theoretical analysis, we give the upper bounds of permanent space of our approach in a worst case. In addition, we report a series of experimental results to validate the correctness and efficiency of the proposed approach.
Efficient Immediate-Access Dynamic Indexing

Alistair Moffat,Joel Mackenzie

DOI: https://doi.org/10.1016/j.ipm.2022.103248

2023-01-11

Abstract:In a dynamic retrieval system, documents must be ingested as they arrive, and be immediately findable by queries. Our purpose in this paper is to describe an index structure and processing regime that accommodates that requirement for immediate access, seeking to make the ingestion process as streamlined as possible, while at the same time seeking to make the growing index as small as possible, and seeking to make term-based querying via the index as efficient as possible. We describe a new compression operation and a novel approach to extensible lists which together facilitate that triple goal. In particular, the structure we describe provides incremental document-level indexing using as little as two bytes per posting and only a small amount more for word-level indexing; provides fast document insertion; supports immediate and continuous queryability; provides support for fast conjunctive queries and similarity score-based ranked queries; and facilitates fast conversion of the dynamic index to a "normal" static compressed inverted index structure. Measurement of our new mechanism confirms that in-memory dynamic document-level indexes for collections into the gigabyte range can be constructed at a rate of two gigabytes/minute using a typical server architecture, that multi-term conjunctive Boolean queries can be resolved in just a few milliseconds each on average even while new documents are being concurrently ingested, and that the net memory space required for all of the required data structures amounts to an average of as little as two bytes per stored posting, less than half the space required by the best previous mechanism.

Information Retrieval,Databases
Bass: Approximate Search on Large String Databases

J Yang,W Wang,P Yu

DOI: https://doi.org/10.1109/ssdbm.2004.20

2004-01-01

Abstract:In this paper we study the problem on how to build an index structure for large string databases to efficiently support various types of string matching without the necessity of mapping the substrings to a numerical space (e.g., string B-tree and MRS-index) nor the restriction of in-memory practice (e.g., suffix tree and suffix array). Towards this goal, we propose a new indexing scheme, BASS-tree, to efficiently support general approximate substring match (in terms of certain symbol substitutions and misalignments) in sublinear time on a large string database. The key idea behind the design is that all positions in each string are grouped recursively into a fully balanced tree according to the similarities of the subsequent segments starting at those positions. Each node is labeled with a regular expression that describes the commonality of the substrings indexed through the subtree. Any search can then be properly directed to the portion in the database with a high potential of matching quickly. With the BASS-tree in place, wild card(s) in the query pattern can also be handled in a seamless way. In addition, search of a long pattern can be decomposed into a series of searches of short segments followed by a process to join the results. It has been demonstrated in our experiments that the potential performance improvement brought by BASS-tree is in an order of magnitude over alternative methods.
Practical Techniques for Fast Searches on Encrypted String Data in Databases

CUI Bin-Ge,LIU Da-Xin,WANG Tong

2006-01-01

Computer Science

Abstract:To solve the problem of querying on encrypted string data in databases,an auxiliary index column is built for each attribute need to be encrypted,which will be used in two-phrase query. The contents of the index column are composed of two different parts. The first part corresponds to the partition value of the encrypted string data,which will be used in the exact match query. The second part corresponds to the feature value of the encrypted string data,which will be used in the fuzzy match query. When querying data,coarse query is performed first on the encrypted data based on the index column,which filtrates those records not related to the query conditions. The remaining records will be decrypted,and refined query is performed on the decrypted data. Results of a series of experiments validate the functionality and usability of the approach.
Adjacency Matrix Based Full-Text Indexing Models

Shuigeng Zhou,Jihong Guan,Yunfa Hu,Jiangtao Hu,Aoying Zhou

DOI: https://doi.org/10.1007/3-540-47714-4_6

2002-01-01

Journal of Software

Abstract:This paper proposes two new character-based full-text indexing models, i.e., adjacency matrix based inverted file and adjacency matrix based PAT array. Formally, the former is a kind of reorganization of the traditional inverted file, and the latter is a kind of decomposition of the traditional PAT array. Both organize text-indexing information in the form of adjacency matrix. Query algorithms for the new models are developed and performance comparisons between the new models and the traditional models are carried out. The new models can improve query-processing efficiency considerably at the cost of much less amount of extra storage overhead compared to the size of original text database, so are suitable for applications of large-scale text databases, especially Chinese text databases.
Scalable Top-K Spatial Keyword Search

Dongxiang Zhang,Kian-Lee Tan,Anthony K. H. Tung

DOI: https://doi.org/10.1145/2452376.2452419

2013-01-01

Abstract:In this big data era, huge amounts of spatial documents have been generated everyday through various location based services. Top- k spatial keyword search is an important approach to exploring useful information from a spatial database. It retrieves k documents based on a ranking function that takes into account both textual relevance (similarity between the query and document keywords) and spatial relevance (distance between the query and document locations). Various hybrid indexes have been proposed in recent years which mainly combine the R-tree and the inverted index so that spatial pruning and textual pruning can be executed simultaneously. However, the rapid growth in data volume poses significant challenges to existing methods in terms of the index maintenance cost and query processing time. In this paper, we propose a scalable integrated inverted index, named I 3 , which adopts the Quadtree structure to hierarchically partition the data space into cells. The basic unit of I 3 is the keyword cell, which captures the spatial locality of a keyword. Moreover, we design a new storage mechanism for efficient retrieval of keyword cell and preserve additional summary information to facilitate pruning. Experiments conducted on real spatial datasets (Twitter and Wikipedia) demonstrate the superiority of I 3 over existing schemes such as IR-tree and S2I in various aspects: it incurs shorter construction time to build the index, it has lower index storage cost, it is order of magnitude faster in updates, and it is highly scalable and answers top- k spatial keyword queries efficiently.
Text Indexing for Long Patterns using Locally Consistent Anchors

Lorraine A. K. Ayad,Grigorios Loukides,Solon P. Pissis

2024-07-16

Abstract:In many real-world database systems, a large fraction of the data is represented by strings: sequences of letters over some alphabet. This is because strings can easily encode data arising from different sources. It is often crucial to represent such string datasets in a compact form but also to simultaneously enable fast pattern matching queries. This is the classic text indexing problem. The four absolute measures anyone should pay attention to when designing or implementing a text index are: (i) index space; (ii) query time; (iii) construction space; and (iv) construction time. Unfortunately, however, most (if not all) widely-used indexes (e.g., suffix tree, suffix array, or their compressed counterparts) are not optimized for all four measures simultaneously, as it is difficult to have the best of all four worlds. Here, we take an important step in this direction by showing that text indexing with sampling based on locally consistent anchors (lc-anchors) offers remarkably good performance in all four measures, when we have at hand a lower bound $\ell$ on the length of the queried patterns -- which is arguably a quite reasonable assumption in practical applications. Our index offers average-case guarantees. In our experiments using real benchmark datasets, we show that it compares favorably based on the four measures to all classic indexes: (compressed) suffix tree; (compressed) suffix array; and the FM-index. Notably, we also present a counterpart of our index with worst-case guarantees based on the lc-anchors notion of partitioning sets. To the best of our knowledge, this is the first index achieving the best of all worlds in the regime where we have at hand a lower bound $\ell$ on the length of the queried patterns.

Data Structures and Algorithms,Databases
Efficient Bulk Loading to Accelerate Spatial Keyword Queries

Dongsheng Li,Jinkun Pan,Jiaxin Li,Kian-Lee Tan,Dongxiang Zhang

DOI: https://doi.org/10.1109/icpads.2013.87

2013-01-01

Abstract:With the fast development of location-based services and geo-tagging, spatial keyword queries that retrieve objects satisfying both spatial and keyword conditions are gaining in prevalence. A hybrid index that integrates a spatial index (e.g., the R-tree or its variations) with a keyword filter offers a promising approach for processing such queries efficiently. However, it is still an open problem on how a hybrid index can be effectively constructed from scratch. The state-of-the-art bulk loading algorithms for the R-tree consider only spatial relationship, and cannot be employed for the hybrid index. In this paper, we propose a new bulk loading algorithm, named TPA, which constructs a hybrid index top-down. TPA utilizes a two-phase method to construct the children of nodes at each level of the hybrid index, taking both spatial and keyword information into consideration, and thus optimizes the hybrid index for spatial keyword queries. We analyze and evaluate its performance using both real and synthetic datasets. Comprehensive experiments show that TPA can achieve good performance and space utilization, reducing the construction time, the query latency and the index size remarkably.

Using Additional Indexes for Fast Full-Text Search of Phrases That Contain Frequently Used Words

About a structure of easily updatable full-text indexes

Proximity Full-Text Search by Means of Additional Indexes with Multi-component Keys: In Pursuit of Optimal Performance

Proximity Full-Text Search with a Response Time Guarantee by Means of Additional Indexes

Relevance ranking for proximity full-text search based on additional indexes with multi-component keys

Proximity full-text searches of frequently occurring words with a response time guarantee

Efficient Phrase Querying with Flat Position Index

Selection of Optimal Parameters in the Fast K-Word Proximity Search Based on Multi-component Key Indexes

Two simple full-text indexes based on the suffix array

An efficient algorithm for three-component key index construction

Can phrase indexing help to process non-phrase queries?

Augmented Keyword Search on Spatial Entity Databases

Processing Long Queries Against Short Text

An Efficient Approach for Building Compressed Full-Text Index for Structured Data

Efficient Immediate-Access Dynamic Indexing

Bass: Approximate Search on Large String Databases

Practical Techniques for Fast Searches on Encrypted String Data in Databases

Adjacency Matrix Based Full-Text Indexing Models

Scalable Top-K Spatial Keyword Search

Text Indexing for Long Patterns using Locally Consistent Anchors

Efficient Bulk Loading to Accelerate Spatial Keyword Queries