Research And Implementation Of Active Defense Technology For Malicious Crawlers

With the continuous progress of big data era,web crawler technology is rapidly evolving as a tool that enables automated,persistent interaction with Internet applications.Web crawlers cause a lot of security risks to the network environment while bringing convenience,such as malicious data collection in batch;matching collection of various types of community sensitive information;non-authorized ticket brushing,comments;all kinds of malicious attacks based on web crawler variants and other malicious acts.The existing web crawler defense mechanism mainly uses passive detection and interception technology to limit data collection and has played a certain protective effect,but the effect of static passive detection has decreased with the continuous iteration of artificial intelligence technology such as verification code recognition,humanoid behavior and other technologies.In addition,there are also great defects in using IP address as the condition of detection and blocking due to the limited of IP address.This paper proposes a malicious crawler active defense technology model based on the above problems and the key points of web crawler technology,which changes the inherent static characteristics of Web pages through dynamic algorithm and combines with the idea of moving target defense in active defense to deal with complex and changeable web crawler and its variant attacks.The main work of this paper is as follows:1.A dynamic algorithm of Web page is proposed.The algorithm uses dynamic encryption algorithm and dynamic trap technology to adaptively dynamically hop Web pages,effectively avoiding the automatic implementation of malicious reptiles and their variants caused by static,oneness and certainty.2.An evaluation model of access traffic anomaly detection based on browser fingerprint technology is proposed.The model quantifies the proportion of malicious access entities in access traffic to measure the security risk of the environment,which provides reliable data support for system early warning and adaptive hopping strategy.3.An adaptive hopping frequency strategy is proposed.The strategy combines the evaluation model of access traffic anomaly detection with the real-time performance of the system,which reduces the energy consumption problem caused by the hopping and ensures the balance of security,availability and energy consumption.4.Finally,a highly available and distributed malicious crawler active defense system is designed and implemented based on the above malicious crawler active defense model.The experimental results verify the availability,security and high performance of the system.