Research and implementation of vertical search engine based on Carrot2 clustering

GAO Kai,YANG Chun-hui,TAO Qiu-hong,ZHANG Yang,YANG Jun-wei

2012-01-01

Abstract:This paper presents the implementation of vertical search engine based on Nutch,mainly the implementation of the Lucene and the Carrot2 for information retrieval and clustering.Moreover,the paper also introduces in some details the Chinese word segmentation and data collection.

What problem does this paper attempt to address?

Research of Vertical Search Engine in News Industry

M. Li,X. J. Gu,Z. X. Yang

DOI: https://doi.org/10.1109/ismot.2012.6679470

2012-01-01

Abstract:In summing up the existing network of reptiles, and full-text retrieval based on theoretical knowledge, conducted a Web crawler optimization algorithm so that it can adapt to the needs of vertical search engines, and then sub-word component of Pango and Lucene.Net build an efficient full-text search functions. The innovation of the paper is the analysis of the characteristics of news sites to integrate its features into the traditional vertical search engines. News site on the information requirements for the characteristics of the network by studying the relevant full-text search framework to multi-threaded data collection and retrieval of the vertical search engine, performance and user experience goals are to achieve a better. The entire system by small and medium news site commissioning tests designed to meet the test show that the crawlers can adapt to the new network news industry efficient and timely collection requirements, Lucene.Net segmentation. The integration of Pango built the content for news and information Full-text retrieval system can achieve the accuracy of search engine queries for information and efficient response time demands, thereby increasing the amount of information and user experience.
Research and Implementation of Subject-Oriented Vertical Search Engine System

SHI Quan,WANG Heng-shan,XIAO Yang-hua,DING Wei-ping

DOI: https://doi.org/10.19304/j.cnki.issn1000-7180.2011.07.001

2011-01-01

Abstract:A general search engine usually suffers from returning to users too much unrelated information.To avoid these problems of general search engine,a subject-oriented vertical search engine is proposed in this paper followed by the introduction of the crawling strategy and the architecture of this vertical search engine.Some distinctive characteristics of this search engine include: the system runs its core modules on multiples threads;a topic-related determination algorithm based on VSM is implemented in the system.The search engine system as well as its core algorithms is built upon lucene.Net.Experimental results show that the accuracy and recall rate of this system are higher than general search engine.Therefore,it is practically valuable and can be applied into more real applications.
Online Multi-keywords Map-search Algorithm Based on Sub-region Method

Zhi Yu,Zhonghan Sun,Yinlong Zhu,Can Wang

DOI: https://doi.org/10.2991/iccset-14.2015.89

2015-01-01

Abstract:Nowadays, the map-search technology is frequently used by people. However, present map-search service can hardlysatisfy people's growing demands. Providing multiplex service support in search service, such as intuitive search results and multi-keywords search, has become a research focus. This paperproposes a novel online multi-keywords map-search algorithm based on sub-region method. This algorithm makes up a graph of separate nodes and transfers the weight of keyword totheir adjacent nodes. When searching for sub-graph of a single keyword, it finds the node with the max weight then tries to accept or rejectits neighbors step by step. Duringthe search ofsub-graphs of multi-keywords, we must find the node with the max harmonic mean weight and union all the sub-graphs of single keyword it belongs to. This algorithm, based on sub-region method, can search multi-keywords in practical map-search experiments, and return several best resultsat the same time.
VSEC: A vertical search engine for e-commerce

Quan Shi,Zhenquan Shi,Yanghua Xiao

DOI: https://doi.org/10.1007/978-3-642-28798-5_9

2012-01-01

Abstract:With the explosion of e-commerce sites, querying information residing on these sites with high accuracy becomes a practical issue when designing next generation search engines. The inherent natures of general search engines limit their applications in scenarios where result quality is a critical concern. To overcome this difficulty, we build a Vertical Search Engine for e-commerce (VSEC), which crawls, deep-processes, index information of e-commerce sites of given topics and provide querying service of high quality. VSEC is characterized by two defining features: (1) multi-thread model is employed to speed up the crawling procedure; (2) index building and query answering is implemented on Lucene.Net. Experimental results show that our system has high retrieval efficiency and higher query accuracy and recall rate than general search engine. © 2012 Springer-Verlag.
TSearch: A Self-learning Vertical Search Spider for Travel.

Suke Li,Zhong Chen,Liyong Tang,Zhao Wang

DOI: https://doi.org/10.1109/icycs.2008.338

2008-01-01

Abstract:A self-learning vertical search spider for travel is presented. This paper focuses on two machine learning methods SNBC (self-learning naive Bayes classifier) and LQNBC (log quotient naive Bayes classifier) for improving search quality and topic relevance. A framework of designing and implementing a vertical spider TSearch with basic general search spider architecture and functions is also showed. TSearch uses SNBC to filter HTML pages and relies on LQNBC to detect unknown travel related Web sites with high precision. The recall and the precision for the classification of texts crawled by TSearch were measured experimentally. These experiments indicate that using LQNBC and SNBC, TSearch can produce promising travel related information for search.
Query Triggered Crawling Strategy: Build a Time Sensitive Vertical Search Engine

Yu Wu,Lidan Shou,Tianlei Hu,Gang Chen

DOI: https://doi.org/10.1109/cw.2008.35

2008-01-01

Abstract:In today's information society, it is important to retrieve fresh information. Many of the vertical search results are valid to users in only a short period of time. But due to resource constraints, it is not possible to keep the entire local storage synchronized with the Web. We implemented a TSVS (time sensitive vertical search engine) prototype named Velocisaurus focused on time-critical airfare discount information search to investigate the time critical requirements of vertical search and proposed a QTC (query triggered crawling) strategy to coordinate the crawling systems by real-time user queries and solve this problem. Experiment shows that QTC driven crawlers significantly improves the freshness of the search results and utilizes the resources more efficiently compared to regular search engine crawlers..
Design and implementation of Chinese search engine based on Lucene and Socket communication

Shulin Cao,Qingsong Lv,Yifan Wang,Qian Yin,Xin Zheng

DOI: https://doi.org/10.3969/j.issn.1000-8519.2017.07.036

2017-01-01

Abstract:Lucene is a full-text search engine toolkit written in Java, an object-oriented, multi-tier package that provides a low-coupling, platform-independent and full-text search engine architecture. The network programming based on streaming Socket is an effective method to design the network communication program. Based on Lucene, this paper designs and implements the Chinese search engine by using Chinese word segmentation technology. At the same time, it uses Java and PHP Socket network programming technology to realize the function of using Lucene service through PHP website.
Sesq: A Model-Driven Method For Building Object Level Vertical Search Engines

Ling Lin,Yukai He,Hang Guo,Ju Fan,Lizhu Zhou,Qi Guo,Gang Li

DOI: https://doi.org/10.1007/978-3-540-87877-3_39

2008-01-01

Abstract:In vertical search engine research, many works have been reported. But most of them focus on its key issues such as crawling, extraction, and query and few of them give a total solution for building a complete vertical search engine from scratch in a systematic method. To address this issue, we propose a model-driven method and its supporting tool SESQ. Based on a user defined ER schema for a target domain, the tool can help to build a complete search engine by integrating tasks of crawling, extraction. data management and query within One Unified framework.
Classifying And Ranking: The First Step Towards Mining Inside Vertical Search Engines

Hang Guo,Jun Zhang,Lizhu Zhou

DOI: https://doi.org/10.1007/978-3-540-74469-6_23

2007-01-01

Abstract:Vertical Search Engines (VSEs), which usually work on specific domains, are designed to answer complex queries of professional users. VSEs usually have large repositories of structured instances. Traditional instance ranking methods do not consider the categories that instances belong to. However, users of different interests usually care only the ranking list in their own communities. In this paper we design a ranking algorithm -ZRank, to rank the classified instances according to their importances in specific categories. To test our idea, we develop a scientific paper search engine-CPaper. By employing instance classifying and ranking algorithms, we discover some helpful facts to users of different interests.
Design and Implementation of Cross-Search Retrieval System

Li Guangjian,Liu Xiaojuan,Huang Yongwen

DOI: https://doi.org/10.3969/j.issn.1000-4254.2006.07.016

2006-01-01

Library journal

Abstract:This paper mainly discusses the design idea of the Cross - Search retrieval system, as well as its architecture and functional modules. The key problems solved in the system are the interconnection of different target system applying different protocols, the unified format transforming of result records from multi - sources and the synchronized retrievals. It has given an ideal solution and implementation for retrieving multi - databases simultaneously.
A Content-Based Image Retrieval System Based on Hadoop and Lucene

Chunhao Gu,Yang Gao

DOI: https://doi.org/10.1109/CGC.2012.33

2012-01-01

Abstract:This paper introduces a content-based image retrieval (CBIR) system based on Hadoop and Lucene. Hadoop is a kind of open source software with powerful parallelization and scalability and gradually becomes a popular technique for processing and storage of big data in recent years. We design and implement the system to overcome performance bottlenecks brought by computing complexity and big amount of data when constructing a CBIR system. We will present our ideas, designs and fruits of the system in this paper.
Multi-level and cross-domain search engine based on graph convolutional neural network

Qiang Li,Li Zhuang,Qiulin Wang,Xiaodong Zhang,Jianghai Chen

DOI: https://doi.org/10.1093/ijlct/ctae059

IF: 2.3

2024-01-01

International Journal of Low-Carbon Technologies

Abstract:Abstract With the advent of the information age, there is a growing demand for multi-level cross-domain information retrieval by users. However, current technological solutions have not yet overcome the dilemma of automatic adjustment of search scope in cross-domain retrieval. This article proposes a multi-level cross-domain retrieval engine based on graph convolutional neural networks (GCNs). GCN can effectively capture the complex semantic structures in documents, excavate the potential dynamic context of documents by learning the relationships between nodes in the graph structure, adapt to cross-domain retrieval, extract semantic information from different levels and automatically adjust the search scope. To verify the effectiveness of the method, experiments were conducted on the NQ320k dataset and compared with SEAL, BM25, Transformer and Bert models. The results show that the cross-domain retrieval engine based on GCN has significant advantages over other methods, with Recall@100 reaching 97.44%, MRR@100 reaching 76.12% and R-Precision reaching 78.84%. In the future, the performance and applicability of the retrieval engine can be further improved by optimizing the model structure and expanding the training dataset.

energy & fuels,thermodynamics
Design and implementation of a vertical search engine for web stock review

Shuai Wang,Yongfeng Huang,Ping Hu

DOI: https://doi.org/10.16157/j.issn.0258-7998.2017.06.030

2017-01-01

Abstract:The stock market is a barometer of the national macroeconomic environment,the views of users on the stock market to a large extent reflect the views of the stock market,but also affect the stock market.Therefore,through the emotional network text polarity analysis technology and search engine technology to mining stock network view is one of the hot topic of current financial information and cross discipline.This paper designed a fusion vertical search full-text search and opinion mining for network stock review engine system structure,put forward a network oriented theme stock data acquisition point harvest algorithm and crawler structure,and established a multi granularity unsupervised emotion network stock review fuzzy calculation polarity classification method,to achieve the analysis of online stock comment.The test results show that the classification accuracy,the efficiency of data acquisition and the response time of search engine can meet the requirements of practical application.
A pattern-based selective recrawling approach for object-level vertical search.

Yaqian Zhou,Qi Zhang,Xuanjing Huang,Lide Wu

DOI: https://doi.org/10.1145/2505515.2505707

2013-01-01

Abstract:Traditional recrawling methods learn navigation patterns in order to crawl related web pages. However, they cannot remove the redundancy found on the web, especially at the object level. To deal with this problem, we propose a new hypertext resource discovery method, called ``selective recrawling'' for object-level vertical search applications. The goal of selective recrawling is to automatically generate URL patterns, then select those pages that have the widest coverage, and least irrelevance and redundancy relative to a pre-defined vertical domain. This method only requires a few seed objects and can select the set of URL patterns that covers the greatest number of objects. The selected set can continue to be used for some time to recrawl web pages and can be renewed periodically. This leads to significant savings in hardware and network resources. In this paper we present a detailed framework of selective recrawling for object-level vertical search. The selective recrawling method automatically extends the set of candidate websites from initial seed objects. Based on the objects extracted from these websites it learns a set of URL patterns which covers the greatest number of target objects with little redundancy. Finally, the navigation patterns generated from the selected URL pattern set are used to guide future crawling. Experiments on local event data show that our method can greatly reduce downloading of web pages while maintaining comparative object coverage.
Design and Implementation of Craweper Based on Scrapy

Xiaoju Ma,Min Yan

DOI: https://doi.org/10.1088/1742-6596/2033/1/012204

2021-09-01

Journal of Physics: Conference Series

Abstract:With the development of the Internet, network data covers various fields, but with the increasing amount of network data and diversified data formats, it becomes more and more difficult for users to obtain valuable data from massive data. At present, data collection technology has been studied at home and abroad, and has found that network resources can be automatically obtained through network crawler technology. This article takes the information of second-hand housing in Chongqing as an example, designs a crawler program based on Scrapy framework, captures and stores the housing information in some central and western regions, and finally uses Excel data analysis to analyze the second-hand housing resources in Chongqing according to the region and house type. The results show that this program can automatically collect housing information from Anju customers, improve the speed and quality of user obtaining information, and provide a data source for user data analysis.
Design and Research of Web Crawler Based on Distributed Architecture

Lili Wang,Haoliang Wang

DOI: https://doi.org/10.1145/3495018.3495061

2021-10-23

Abstract:Internet data is abundance in content and diverse in organization. In order to automatically complete the process of collecting, analyzing, and storing large amounts of data and information on the web, a web crawler technology based on Hadoop distributed clusters is proposed. Using Nutch crawler framework, Hadoop distributed technology, and Zookeeper distributed coordination service framework, through the construction of distributed clusters, and high-performance Key-Value database Redis to store data, it verifies the feasibility of distributed crawlers. Through the analysis of data collection experiments, the comparison of multiple sets of experimental data between the distributed crawler based on Nutch and the traditional crawler shows that the crawler design of the distributed architecture is superior to the traditional crawler in terms of collection speed and efficiency.
Design and Implementation of Web-based Full-text Search Engine

CHEN Kang,XU Ting,DAI Wenjun,WU Gangshan

DOI: https://doi.org/10.3969/j.issn.1000-3428.2005.20.020

2005-01-01

Abstract:This paper introduces the design and implementation of a Web-based full-text search engine.It gives the design of the system’s framework and the implementation of function modules in detail,and in the end,it gives the test result of the system.
Object-level Vertical Search.

Zaiqing Nie,Ji-Rong Wen,Wei-Ying Ma

2007-01-01

Abstract:Current web search engines essentially conduct document-level ranking and retrieval. However, structured information about realworld objects embedded in static webpages and online databases exists in huge amounts. We explore a new paradigm to enable web search at the object level in this paper, extracting and integrating web information for objects relevant to a specific application domain. We then rank these objects in terms of their relevance and popularity in answering user queries. In this paper, we introduce the overview and core technologies of object-level vertical search engines that have been implemented in two working systems: Libra Academic Search (http://libra.msra.cn) and Windows Live Product Search (http://products.live.com).
Architecture and implementation of an object-level vertical search

Jianfeng Zheng,Zaiqing Nie

DOI: https://doi.org/10.1109/NISS.2009.22

2009-01-01

Abstract:Current mobile web search engines essentially conduct document-level ranking and retrieval. However, structured information about real-world objects embedded in static WebPages and online databases exists in huge amounts. We explore a new paradigm to enable web search at the object level in this paper, extracting and integrating web information for objects relevant to a specific application domain. We then rank these objects in terms of their relevance and popularity in answering user queries. We introduce the overview and core technologies of object-level vertical search engines that have been implemented in one working systems: Windows Live Product Search on mobile (http://m.live.com). © 2009 IEEE.
The Implementation of Hadoop-based Crawler System and Graphlite-based PageRank-Calculation In Search Engine

Qingpei Guo,Chao Xu,Yang Song

DOI: https://doi.org/10.48550/arXiv.1506.00130

2015-05-30

Abstract:Nowadays, the size of the Internet is experiencing rapid growth. As of December 2014, the number of global Internet websites has more than 1 billion and all kinds of information resources are integrated together on the Internet, however,the search engine is to be a necessary tool for all users to retrieve useful information from vast amounts of web data. Generally speaking, a complete search engine includes the crawler system, index building systems, sorting systems and retrieval system. At present there are many open source implementation of search engine, such as lucene, solr, katta, elasticsearch, solandra and so on. The crawler system and sorting system is indispensable for any kind of search engine and in order to guarantee its efficiency, the former needs to update crawled vast amounts of data and the latter requires real-time to build index on newly crawled web pages and calculae its corresponding PageRank value. It is unlikely to accomplish such huge computation tasks depending on a single hardware implementation of the crawler system and sorting system,from which aspect, the distributed cluster technology is brought to the front. In this paper, we use the hadoop Map - Reduce computing framework to implement a distributed crawler system, and use the GraphLite, a distributed synchronous graph-computing framework, to achieve the real-time computation in getting the PageRank value of the new crawled web page.

Distributed, Parallel, and Cluster Computing

Research and implementation of vertical search engine based on Carrot2 clustering

Research of Vertical Search Engine in News Industry

Research and Implementation of Subject-Oriented Vertical Search Engine System

Online Multi-keywords Map-search Algorithm Based on Sub-region Method

VSEC: A vertical search engine for e-commerce

TSearch: A Self-learning Vertical Search Spider for Travel.

Query Triggered Crawling Strategy: Build a Time Sensitive Vertical Search Engine

Design and implementation of Chinese search engine based on Lucene and Socket communication

Sesq: A Model-Driven Method For Building Object Level Vertical Search Engines

Classifying And Ranking: The First Step Towards Mining Inside Vertical Search Engines

Design and Implementation of Cross-Search Retrieval System

A Content-Based Image Retrieval System Based on Hadoop and Lucene

Multi-level and cross-domain search engine based on graph convolutional neural network

Design and implementation of a vertical search engine for web stock review

A pattern-based selective recrawling approach for object-level vertical search.

Design and Implementation of Craweper Based on Scrapy

Design and Research of Web Crawler Based on Distributed Architecture

Design and Implementation of Web-based Full-text Search Engine

Object-level Vertical Search.

Architecture and implementation of an object-level vertical search

The Implementation of Hadoop-based Crawler System and Graphlite-based PageRank-Calculation In Search Engine