Focused Crawling Algorithm for BBS Information Retrieval

LIU Zuo-da,ZHANG Jiu-ling,CHEN Mao-ke,LI Xing

2010-01-01

Abstract:The architecture of BBS is analyzed.And a suitable focused crawling algorithm is proposed,which improves the efficiency.A BBS search engine is designed based on the proposed algorithm.Practical experiments have been carried out,which show that the recall ratio as well as the accuracy of searching has been enhanced.

What problem does this paper attempt to address?

New Focused Crawling Algorithm

Su Guiyang,Li Jianhua,Ma Yinghua,Li Shenghong,Song JupingDepartment of Electronic Engineering,Shanghai Jiaotong University,Shanghai,P. R. China

IF: 1.363

2005-01-01

Journal of Systems Engineering and Electronics

Abstract:Focused carawling is a new research approach of search engine. It restricts information retrieval and provides search service in specific topic area. Focused crawling search algorithm is a key technique of focused crawler which directly affects the search quality. This paper first introduces several traditional topic-specific crawling algorithms, then an inverse link based topic-specific crawling ...
URL Rule Based Focused Crawler

Xiaolin Zheng,Tao Zhou,Zukun Yu,Deren Chen

DOI: https://doi.org/10.1109/icebe.2008.61

2008-01-01

Abstract:Vertical search engines use focused crawlers as their key component and develop some specific algorithms to select Web pages relevant to some pre-defined set of topics. Therefore, how to effectively build up a semantic pattern for specific topics is extremely important to such search engines. In this paper we propose an UBFC (URL rule based focused crawler) algorithm based on a double-crawler framework (an experimental crawler and a focused crawler). The kernel of our UBFC is an URL regular expression learner, which is used to automatically learn and generalize the regular expressions of URLs of the sample Web pages. The so generated URL regular expressions are used to direct the UBFC to work. Using UBFC, we developed a Nutch based focused crawler for hardwaretoday.com and get an excellent result. To evaluate the UBFC, we conduct some experiments to make an analysis through comparison with two proposed methods, the BFSC (breadth first search crawler) and BLFC (baseline focused crawler).
Focused Crawler Framework Based On Open Search Engine

Jiawei Liu,Yongfeng Huang

DOI: https://doi.org/10.1007/978-3-030-00012-7_6

2018-01-01

Abstract:When users need to analyze webpages related to some specific topics, generally they use crawlers to acquire webpages, and then analyze the results to extract those match the users' interests. However, in data acquisition stage, users usually have customize demand on acquiring data. Ordinary crawler systems are very resource-constrained so they cannot traverse the entire internet. Meanwhile, search engines can satisfy these demand but it relies on many manual interactions. The traditional solution is to constrain the crawlers in some limited domain, but this will lead to the problem of low recall rate as well as inefficiency. In order to solve the problems above, this paper does some research on focused crawlers framework based on open search engine. It takes advantage of open search engine's information gather and retrieval capabilities, and can automatically/ semi-automatically generate the topic model to interpret and complete users search intents, with only a few seed keywords need to be provided initially. Then it uses open search engine interfaces to iteratively crawl topic-specific webpages. Compared with the traditional ways, the focused crawler based on open search engine proposed in this paper improves the recall rate and efficiency under the premise of ensuring the accuracy.
Survey on the research of focused crawling technique

Li-Zhu ZHOU,Ling LIN

2005-01-01

Journal of Computer Applications

Abstract:The survey of focused crawling starts with the motivation for this new research and an introduction on basic concepts of focused crawling.The key issues in focused crawling are reviewed,such as webpage analyzing algorithms and the searching strategy on the Web.How to crawl relevant data and information according to different requirements is discussed in detail and three representative architectures of focused crawler systems are analyzed.Some future works for focused crawling research are indicated,including crawling for data analysis and data mining,topic description,finding relevant Web pages,Web data cleaning,and the extension of search space.
Focused Crawling Using Navigational Rank.

Shicong Feng,Li Zhang,Yuhong Xiong,Conglei Yao

DOI: https://doi.org/10.1145/1871437.1871660

2010-01-01

Abstract:The goal of focused crawling is to use limited resources to effectively discover web pages related to a specific topic rather than downloading all accessible web documents. The major challenge in focused crawling is how to effectively determine each hyperlink's capability of leading to target pages. To compute this capability, we present a novel approach, called Navigational Rank (NR) . In general, NR is a kind of two-step and two-direction credit propagation approach. Compared to existing methods, NR mainly has three advantages. First, NR is dynamically updated during the crawling progress, which can adapt to different website structures very well. Second, when the crawling seed is far away from the target pages, and the target pages only constitute a small portion of the whole website, NR shows a significant performance advantage. Third, NR computes each link's capability of leading to target pages by considering both the target and non-target pages it leads to. This global knowledge causes a better performance than only using target pages. We have performed extensive experiments for performance evaluation of the proposed approach using two groups of large-scale, real-world datasets from two different domains. The experimental results show that our approach is domain-independent and significantly outperforms the state-of-arts.
A Semantic and Optimized Focused Crawler Based on Semantic Graph and Genetic Algorithm

Wenhao Huang,Xiaoyan Li,Xiao Zhou,Deyu Qi,Jianqing Xi,Wenjun Liu,Feiyu Zhao

DOI: https://doi.org/10.3390/sym16111439

2024-10-31

Symmetry

Abstract:A focused crawler automatically retrieves, organizes, and extracts specific topic-related information from the internet for analysis and application. Currently, most focused crawlers assess the relevance of web pages to a given topic through methods such as keyword matching, semantic analysis, and link structures. However, these existing focused crawlers suffer from issues such as misleading directions and reduced accuracy due to the lack of semantic analysis of topic terms, as well as biased computation of topic relevance caused by the absence of effective weighting factors. To solve the above-mentioned problems, this study proposes a semantic and optimized focused crawler based on Semantic Graph and Genetic Algorithm. The proposed crawler eliminates ambiguous terms by constructing a semantic graph, optimizes the weighting factors of topic relevance with asymmetry by using a genetic algorithm, and combines both above two points to predict the priority of each unvisited hyperlink. The experiment results indicate that the proposed SG-GA Crawler improves the evaluation indicators compared with the other three focused crawlers, including VSM Crawler, SSRM Crawler, and SG Crawler. More specifically, the percentage improvement achieved by the proposed method exceeds 19%, 19%, and 13% in terms of three evaluation indicators, including the number of relevant web pages, acquisition rate, and average relevance, respectively. In conclusion, the proposed focused crawler can grab more quantity and higher quality topic-related web pages from the Internet.

multidisciplinary sciences
An article language model for BBS search

Jingfang Xu,Yangbo Zhu,Xing Li

DOI: https://doi.org/10.1007/11531371_22

2005-01-01

Abstract:Bulletin Board Systems (BBS), similar to blogs, newsgroups, online forums, etc., are online broadcasting spaces where people can exchange ideas and make announcements. As BBS are becoming valuable repositories of knowledge and information, effective BBS search engines are required to make the information universally accessible and useful. However, the techniques that have been proven successful for web search are not suitable for searching BBS articles due to the nature of BBS. In this paper, we propose a novel article language model (LM) to build an effective BBS search engine. We investigate the differences between BBS articles and web pages, then extend the traditional LM to author LM and category LM. The article LM is powerful in the sense that it can combine the three LMs into a single framework. Experimental results shows that our article LM substantially outperforms both INQUERY algorithm and the traditional LM.
New algorithm of topic-oriented crawler

LI Wei-jiang,ZHAO Tie-jun,PIAO Xing-hai

2009-01-01

Abstract:The general crawler provides more help to people for finding information in WWW.However,it has some drawback in terms of precision and efficiency because of its generality and no specialty.This paper addressed two issues of the topic-oriented Web crawler.One is how to make the definition of the topic,the other is how to sort of links to be downloaded in the queue efficiently.It aimed to visit only relevant pages,and got a great scale of hyperlinks which link to the relevant pages.The crawl method is a novel one,which was based on the semi-structured features of the website and content information.The results of experiment show that it is a very effective method for focused crawler.
Analysis and Evaluation of the Link and Content Based Focused Treasure-Crawler

Ali Seyfi

DOI: https://doi.org/10.1016/j.csi.2015.09.007

2013-06-01

Abstract:Indexing the Web is becoming a laborious task for search engines as the Web exponentially grows in size and distribution. Presently, the most effective known approach to overcome this problem is the use of focused crawlers. A focused crawler applies a proper algorithm in order to detect the pages on the Web that relate to its topic of interest. For this purpose we proposed a custom method that uses specific HTML elements of a page to predict the topical focus of all the pages that have an unvisited link within the current page. These recognized on-topic pages have to be sorted later based on their relevance to the main topic of the crawler for further actual downloads. In the Treasure-Crawler, we use a hierarchical structure called the T-Graph which is an exemplary guide to assign appropriate priority score to each unvisited link. These URLs will later be downloaded based on this priority. This paper outlines the architectural design and embodies the implementation, test results and performance evaluation of the Treasure-Crawler system. The Treasure-Crawler is evaluated in terms of information retrieval criteria such as recall and precision, both with values close to 0.5. Gaining such outcome asserts the significance of the proposed approach.

Information Retrieval,Digital Libraries
A New Architecture of an Intelligent Agent-Based Crawler for Domain-Specific Deep Web Databases

Yanni Li,Yuping Wang,Erfang Tian

DOI: https://doi.org/10.1109/wi-iat.2012.103

2012-01-01

Abstract:A key problem of retrieving, integrating and mining rich and high quality information from massive Deep Web Databases (WDBs) online is how to automatically and effectively discover and recognize domain-specific WDBs' entry points, i.e., searchable forms, in the Web. It has been a challenging task because domain-specific WDBs' forms with dynamic and heterogeneous properties are very sparsely distributed over several trillion Web pages. Although significant efforts have been made to address the problem and its special cases, more intelligent and effective solutions remain to be further explored. In this paper, a new architecture of an intelligent agent-based crawler (iCrawler) for domain-specific Deep Web databases has been proposed to address the limitations of the existing methods. The iCrawler, based on intelligent learning agents and domain ontology, and a series of novel and effective strategies, including a two-step page classifier, a link scoring strategy, etc, can improve the performance of the existing methods. Experiments of the iCrawler over a number of real Web pages in a set of representative domains have been conducted and the results show that the iCrawler outperforms the existing domain-specific Deep Web Form-Focused Crawlers (FFCs) in terms of the harvest rate, coverage rate and time performance.
Focused Crawler Research for Business Intelligence Acquisition

Peng Xin,Qin Qiuli

DOI: https://doi.org/10.14257/ijhit.2013.6.6.16

2013-01-01

International Journal of Hybrid Information Technology

Abstract:The internet has become indispensable part of people's life. For enterprises, there are mass of valuable information in the internet. It not only includes competitor information, but also includes customer’s evaluation of products. These information is an important source of business intelligence. This paper aims to build a focused crawler to filter business intelligence from vast amounts of information in the internet. The crawler takes a certain number of web pages as seed. Then extract URLs in these pages, and parse main text of every URL. After that, the crawler calculates relevancy between every main text and the crawler’s topic based on VSM (vector space model) and TF-IDF (Term Frequency-Inverse Document Frequency). If a web page is relevant, it will be saved; otherwise, it will be discarded. At last, an experiment is done to test the performance of crawler. It can be seen that the recall rate and accuracy of the crawler is very high though the result of this experiment.
A novel focused crawler combining Web space evolution and domain ontology

Jingfa Liu,Xin Li,Qiansheng Zhang,Guo Zhong

DOI: https://doi.org/10.1016/j.knosys.2022.108495

2022-05-01

Abstract:In many fields, how to catch the related-topic Web resources is crucial. As a vertical search method, focused crawler has received great attention in recent years. Currently, most focused crawlers consider multiple evaluating factors of the hyperlinks and use the weighted sum approach to compute the priorities of unvisited hyperlinks. However, the proper weighted coefficients are hard to determine, and their unsuitable values may even cause the direction of crawlers to deviate seriously from the topic. To overcome this issue, this article builds a multi-objective optimization model based on Web text and link structure and designs a crawler framework called the Web space evolution (WSE), where a hyperlink bank whose radius is gradually increased is introduced to extend the search scape of crawlers in Web space. To improve the uniformity and diversity of hyperlinks, a nearest and farthest candidate solution method is combined with the fast non-dominated sorting to choose Pareto-optimal solutions (hyperlinks). A domain ontology based on the formal concept analysis is applied to establish the topic model. By incorporating the WSE and the domain ontology into the focused crawling, a novel focused crawler called FCWSEO is proposed to collect topic-relevant webpages. The experimental results on the rainstorm disaster domain show that the FCWSEO outperforms other focused crawler strategies in terms of the quantity and quality of retrieved relevant webpages.

computer science, artificial intelligence
Designing Focused Crawler Based on Improved Genetic Algorithm

Wei Yan,Li Pan

DOI: https://doi.org/10.1109/icaci.2018.8377476

2018-01-01

Abstract:As focused crawler intends to search the Internet conform to a specific topic, it has a good prospect in the field of vertical search. A good search strategy is the core to improve the accuracy and coverage of focused crawler. Best-First search strategy is often applied but easily falls into local optimization. In order to improve the global search capability, this paper proposes a focused crawler based on improved genetic algorithm. In this algorithm, fitness function considers both topic correlation and importance. Topic correlation is analyzed by vector space model and topic importance is calculated by improved PageRank algorithm. Genetic operations are optimized based on user browsing behavior. Selection operation chooses webpages with high fitness, crossover operation sorts links by topic importance and mutation operation searchs combined keywords through search engine. Compared with existing genetic algorithms, the experimental results show that improved genetic algorithm can enhance precision and recall of focused crawler and enlarge the search scope of the crawler.
Guide Focused Crawler Efficiently and Effectively Using On-Line Topical Importance Estimation

Ziyu Guan,Can Wang,Chun Chen,Jiajun Bu,Junfeng Wang

DOI: https://doi.org/10.1145/1390334.1390488

2008-01-01

Abstract:Focused crawling is a critical technique for topical resource discovery on the Web. We propose a new frontier prioritizing algorithm, namely, the OTIE (On-line Topical Importance Estimation) algorithm, which efficiently and effectively combines link-based and content-based analysis to evaluate the priority of an uncrawled URL in the frontier. We then demonstrate OTIE's advantages over traditional prioritizing algorithms by real crawling experiments.
Customized focused crawler for peer-to-peer Web search

Fang Qiming,Yang Guangwen,Wu Yongwei,Zhu Anping,Zheng Weimin

DOI: https://doi.org/10.3321/j.issn:1671-4512.2007.z2.039

2007-01-01

Abstract:A customized focused crawler is proposed to meet the higher demands of peer-to-peer(P2P) Web search.The crawler employs a simple topic description method for convenient customization and uses link navigation based on link structure of Web sites to achieve efficient crawling.The peer can easily customize a lightweight focused crawler with less resource consumption,higher data crawling precision and better manageability by setting a configuration file.Furthermore,a novel data collection updating mechanism for the crawler is presented.This mechanism is a hybrid of incremental update and batch update,which can reduce the complexity of incremental update and have lower overhead compared with batch update.Experimental results indicate that this hybrid updating mechanism can obtain high freshness and recall.
A New Algorithm of Topical Crawler

Li Wei-jiang,Ru Hua-suo,Zhao Tie-jun,Zang Wen-mao

DOI: https://doi.org/10.1109/wcse.2009.706

2009-01-01

Abstract:The generic crawler provides more help to people for finding information in WWW. However, it has some drawback in terms of precision and efficiency because of its generality and no specialty. In this paper, we address two issues of the topical web crawler. One is how to make the definition of the topic; the other is how to sort of links to be downloaded in the queue efficiently. It aims to visit only relevant pages, and get a great scale of hyperlinks which link to the relevant pages. The crawl method in this paper is a novel one, which is based on the semi-structured features of the website and content information. The results of experiment show that it is a very effective method for focused crawler.
An Efficient Adaptive Focused Crawler Based on Ontology Learning

C Su,Y Gao,JM Yang,B Luo

DOI: https://doi.org/10.1109/ichis.2005.19

2005-01-01

Abstract:The enormous growth of the world wide web in recent years has made it important to perform resource discovery efficiently. Consequently, several new ideas have been proposed; among them a key technique is focused crawling which is able to crawl particular topical portions of the world wide web quickly without having to explore all web pages. In this paper we present an intelligent focused crawler algorithm in which we embeds ontology to evaluate the page's relevance to the topic. Compared with other algorithms using domain knowledge, our algorithm can evolve the ontology automatically during crawl process. Considering the instinct characteristics of the ontology, propagation has also been imported to accelerate the evolution of the ontology. We applied our approaches in several tasks and provided an empirical evaluation which has shown promising results.
UCrawler: A learning-based web crawler using a URL knowledge base

Wei Wang,Lihua Yu

DOI: https://doi.org/10.3233/jcm-204658

2021-05-06

Journal of Computational Methods in Sciences and Engineering

Abstract:Focused crawlers, as fundamental components of vertical search engines, focus on crawling the web pages related to a specific topic. Existing focused crawlers commonly suffer from the problems of low efficiency of crawling pages and subject migration. In this paper, we propose a learning-based focused crawler using a URL knowledge base. To improve the accuracy of similarity, the similarity of the topic is measured with the parent page content, anchor information, and URL content. The URL content is also learned and updated iteratively and continuously. Within the crawler, we implement a crawling mechanism based on a combination of content analysis and simple link analysis crawler strategy, which decreases computational complexity and avoids the locality problem of crawling. Experimental results show that our proposed algorithm achieves a better precision than traditional methods including the shark-search and best-first search algorithms, and avoids the local optimum problem of crawling.
Research and Implementation of Subject-Oriented Vertical Search Engine System

SHI Quan,WANG Heng-shan,XIAO Yang-hua,DING Wei-ping

DOI: https://doi.org/10.19304/j.cnki.issn1000-7180.2011.07.001

2011-01-01

Abstract:A general search engine usually suffers from returning to users too much unrelated information.To avoid these problems of general search engine,a subject-oriented vertical search engine is proposed in this paper followed by the introduction of the crawling strategy and the architecture of this vertical search engine.Some distinctive characteristics of this search engine include: the system runs its core modules on multiples threads;a topic-related determination algorithm based on VSM is implemented in the system.The search engine system as well as its core algorithms is built upon lucene.Net.Experimental results show that the accuracy and recall rate of this system are higher than general search engine.Therefore,it is practically valuable and can be applied into more real applications.
A Focused Crawler For Borderlands Situation Information With Geographical Properties Of Place Names

Dongyang Hou,Hao Wu,Jun Chen,Ran Li

DOI: https://doi.org/10.3390/su6106529

IF: 3.9

2014-01-01

Sustainability

Abstract:Place name is an important ingredient of borderlands situation information and plays a significant role in collecting them from the Internet with focused crawlers. However, current focused crawlers treat place name in the same way as any other common keyword, which has no geographical properties. This may reduce the effectiveness of focused crawlers. To solve the problem, this paper firstly discusses the importance of place name in focused crawlers in terms of location and spatial relation, and, then, proposes the two-tuple-based topic representation method to express place name and common keyword, respectively. Afterwards, spatial relations between place names are introduced to calculate the relevance of given topics and webpages, which can make the calculation process more accurately. On the basis of the above, a focused crawler prototype for borderlands situation information collection is designed and implemented. The crawling speed and F-Score are adopted to evaluate its efficiency and effectiveness. Experimental results indicate that the efficiency of our proposed focused crawler is consistent with the polite access interval and it could meet the daily demand of borderlands situation information collection. Additionally, the F-Score value of our proposed focused crawler increases by around 7%, which means that our proposed focused crawler is more effective than the traditional best-first focused crawler.

Focused Crawling Algorithm for BBS Information Retrieval

New Focused Crawling Algorithm

URL Rule Based Focused Crawler

Focused Crawler Framework Based On Open Search Engine

Survey on the research of focused crawling technique

Focused Crawling Using Navigational Rank.

A Semantic and Optimized Focused Crawler Based on Semantic Graph and Genetic Algorithm

An article language model for BBS search

New algorithm of topic-oriented crawler

Analysis and Evaluation of the Link and Content Based Focused Treasure-Crawler

A New Architecture of an Intelligent Agent-Based Crawler for Domain-Specific Deep Web Databases

Focused Crawler Research for Business Intelligence Acquisition

A novel focused crawler combining Web space evolution and domain ontology

Designing Focused Crawler Based on Improved Genetic Algorithm

Guide Focused Crawler Efficiently and Effectively Using On-Line Topical Importance Estimation

Customized focused crawler for peer-to-peer Web search

A New Algorithm of Topical Crawler

An Efficient Adaptive Focused Crawler Based on Ontology Learning

UCrawler: A learning-based web crawler using a URL knowledge base

Research and Implementation of Subject-Oriented Vertical Search Engine System

A Focused Crawler For Borderlands Situation Information With Geographical Properties Of Place Names