Technology Study On Anomaly Detection Based On WEB Access Log

Under the promotion of the Internet technology improvements, the Internet continues to expand its influence as a medium for communications and commerce, the Internet along with the corporate network even plays an extremely important role in creating and advancing new business avenues. Business needs have motivated enterprises and governments across the globe to develop sophisticated, complex information networks. Such networks incorporate a diverse array of technologies and Web service. However, with the hacker technology progress and system security protection insufficiency results in network security incidents occur frequently in recent two years, the number of network attack is growing bigger. Network attacks detection has become a topic of concern, the research of network attacks detection and prevention needs more in-depth expansion.Today’s commercially available intrusion detection systems are predominantly signature-based intrusion detection systems that are designed to detect known attacks by utilizing the signatures of those attacks. Such systems require frequent rule-base updates and signature updates, and are not capable of detecting unknown attacks and taking much time. The lack of linkage mechanism between most detection systems department internal servers, and lack of effective capacity analysis of mass data make data analysis difficult and time consuming increases ceaselessly. Therefore, there are many researches on anomaly detection of application layer. The technique analyses the characteristic mode of normal behavior and construct the positive security model. This kind of technology can effectively identify the malicious attacks, because it is not based on expert knowledge construction. Therefore, these researches are significant for mining unknown attack methods, new vulnerabilities and tracking new Trojan horse.The work in this paper is based on a project called "WEB Security Model Investigation Based on Massive Log Data" which proposed "WEB Log Intelligent Analysis System for Security". This system analyses the abnormal behaviors by establishing a multi-level model of WEB behaviors, mining potential relationship in the massive log data. This paper introduces and studies various types of anomaly detection methods. Discusses the advantages and disadvantages of those methods and systems, and then find out user behavior pattern using the idea of data mining. Realize two functional modules named IP/UA (User Agent) feature analysis and frequent pattern mining of the log data analysis subsystem, finally identify the abnormal behavior through the feature detection, and classify the abnormality.In order to meet the needs of project, the team builds a massive data analysis platform based on Hadoop and ELK system. The platform collects historical WEB access logs from a large website, using the popular Elastic Search platform to achieve log data storage, query and management and using distributed programming model to process the data to achieve batch analysis and real-time analysis. The work in this paper mainly analyses historical data, test data is log data of August 2013, test module are IP/UA feature analysis module and frequent pattern mining module. The results show that:the Hadoop distributed system can effectively complete the log data processing and significantly shorten the treatment time. IP/UA feature analysis is able to identify users that counterfeit famous crawler, use suspicious UAs or have suspicious characters in UA string. Frequent pattern mining finally comes out four types of abnormal patterns. They can be used for distinguishing whether a new user is abnormal and which type the anomalies is.