Abstract:Motivation: Life stories of diseased and healthy individuals are abundantly available on the Internet. Collecting and mining such online content can offer many valuable insights into patients' physical and emotional states throughout the pre-diagnosis, diagnosis, treatment and post-treatment stages of the disease compared with those of healthy subjects. However, such content is widely dispersed across the web. Using traditional query-based search engines to manually collect relevant materials is rather labor intensive and often incomplete due to resource constraints in terms of human query composition and result parsing efforts. The alternative option, blindly crawling the whole web, has proven inefficient and unaffordable for e-health researchers. Results: We propose a user-oriented web crawler that adaptively acquires user-desired content on the Internet to meet the specific online data source acquisition needs of e-health researchers. Experimental results on two cancer-related case studies show that the new crawler can substantially accelerate the acquisition of highly relevant online content compared with the existing state-of-the-art adaptive web crawling technology. For the breast cancer case study using the full training set, the new method achieves a cumulative precision between 74.7 and 79.4% after 5 h of execution till the end of the 20-h long crawling session as compared with the cumulative precision between 32.8 and 37.0% using the peer method for the same time period. For the lung cancer case study using the full training set, the new method achieves a cumulative precision between 56.7 and 61.2% after 5 h of execution till the end of the 20-h long crawling session as compared with the cumulative precision between 29.3 and 32.4% using the peer method. Using the reduced training set in the breast cancer case study, the cumulative precision of our method is between 44.6 and 54.9%, whereas the cumulative precision of the peer method is between 24.3 and 26.3%; for the lung cancer case study using the reduced training set, the cumulative precisions of our method and the peer method are, respectively, between 35.7 and 46.7% versus between 24.1 and 29.6%. These numbers clearly show a consistently superior accuracy of our method in discovering and acquiring user-desired online content for e-health research.

[Application of Python Web Crawler Technology in Infodemiology].

The Application Framework of Big Data Technology During the COVID-19 Pandemic in China.

Notifiable Infectious Disease Surveillance with Data Collected by Search Engine

Innovative Application of Python in Data Crawling —Chinese Version of Movie Recommendation Platform

Implementation of Web Data Mining Technology Based on Python

Application of Web Crawler Technology Based on Python in Big Data Environment

Design and Implementation of Crawler Program Based on Python

Image Information Collection System Based on Python Web Crawler Technology

Development of automatic collection system of global epidemic informations

Employment Data Analysis based on Python Crawler Technology

Implementation of Recruitment Website Data Analysis System Based on Web Crawler

PYTHON-POWERED DATA ANALYSIS THROUGH WEB SCRAPING

Design and Implementation of Craweper Based on Scrapy

The Web Crawler Based on Python to Obtain Public Emotional data Analysis - Take The Weibo Hot Rearch data as An Example

Data Analysis by Web Scraping using Python

Establishing a Web-Based Integrated Surveillance System for Early Detection of Infectious Disease Epidemic in Rural China: a Field Experimental Study

Python based IT industry recruitment data automatic collection, warehousing, and analysis system

[Progress in application of machine learning in epidemiology]

Research and Application of Artificial Intelligence and Big Data in Infectious Disease Prevention and Control

A user-oriented web crawler for selectively acquiring online content in e-health research.

Data Crawling and Research Based on Topic Web Crawler