Design of Crawler and Visual Interactive Interface Based on Scrapy Framework

Lili Gao,Qingbin Meng
DOI: https://doi.org/10.1109/ICETCI57876.2023.10176505
2023-05-26
Abstract:Nowadays, when studying and analyzing problems, the previous conventional analysis methods are obviously difficult to meet our needs in such a huge data scale environment. At the same time, the emergence of web crawlers solves the problem of data acquisition under big data. This paper completes a series of tasks such as data acquisition, download and storage in the designated website by studying the scrapy framework. At the same time, through the research of pyecharts library, JSON library and other visualization related technologies, we can realize the visual processing and display of the crawling data of the scrapy framework, so as to realize the more intuitive analysis of the data.
Computer Science
What problem does this paper attempt to address?