Research on an anti-crawling mechanism and key algorithm based on sliding time window

Yi Liu,Zhengqiu Yang,Jiapeng Xiu,Chen Liu
DOI: https://doi.org/10.1109/CCIS.2016.7790257
2016-01-01
Abstract:Inadequate crawling behavior of the crawler will have a very serious impact on the site, so anti crawling mechanism is an important function for the website. Most of the existing anti crawling methods are non real time detection, and the recognition accuracy is low. By analyzing the characteristics of Crawler, a real-time crawler detection method based on sliding time window is proposed, which improves the accuracy and efficiency of detection of non compliance with the rules of Crawler.
What problem does this paper attempt to address?