Research On Cloud Security Log Analysis Technology Based On Hadoop

With the advent of the era of the Internet,network security incidents emerge one after another,especially in the era of big data.Network security is particularly important.As an important branch of big data,log data is an important way to record the running status of devices.The security auditing method,which can be used for pre-protection and after-accusation,can grasp the network security situation in real time,and formulate a series of practical and feasible security defense programs to improve network security.The technology of log mining is endless.Wiith the continuous development of the network,clustering analysis is more widely used in data mining in the face of new or no attack patterns.Therefore,the cluster-based log analysis method is In the face of the ever-changing network environment,it has important research value and significance.In addition,security logs,as an auditing tool for recording the operation of network security devices,are getting more and more people's attention,even called "dark data",and their value is not In other words,log analysis in stand-alone mode is more advantageous when dealing with small data sets,but it is stretched when faced with daily data growth at the PB level.To this end,how to store and analyze massive security logs has become the focus of current log analysis.In view of the above problems,this paper makes an in-depth research on the analysis technology of massive security logs based on Hadoop environment.Based on the existing research background and current situation,the clustering classic algorithm K-Means is optimized and The parallelization was realized,and the parallel analysis of massive log was realized.Finally,a log analysis and accountability system was designed and its related performance was tested.The specific work of this paper is as follows:1.Through the importance and status quo of log analysis in current network security,this paper expounds the research background and significance of log clustering analysis based on Hadoop.By summarizing the related work of predecessors,the paper gives the structure of the paper.And the main work and innovation of this article.2.The key technologies in the storage and analysis of massive security logs based on Hadoop are introduced in detail.It mainly includes the foundation of log analysis,the process of log mining,technology,and important core components HDFS and Map/Reduce in Hadoop.3.Aiming at the defects of traditional K-Means in finding the initial clustering center,this paper improves the algorithm and overcomes the defect of randomness when looking for the initial clustering center point.The main idea is the combination of maximum and minimum distances and means.4.For the improved K-Means clustering algorithm,there are many iterative calculations,which affect the clustering efficiency,especially in the face of massive security logs.The improved K-Means algorithm is parallelized,mainly through two Parallelization of the secondary Map/Reduce greatly improves the efficiency of clustering.5.Based on the previous research,combined with the background of log big data environment,a Hadoop-based log analysis and recovery prototype system was researched and designed.Based on the system requirements analysis,the system framework was built and the related system was further designed.The function module mainly includes functions such as log collection,storage,cluster association analysis,display,etc.The user can view the abnormal log through the web interface,and can implement the security control of the ex-post supervision after the event according to the generated evidence chain.6.Test the log analysis and accountability system through key performance evaluation indicators,including cluster and stand-alone performance,acceleration ratio,and log collection and processing capabilities to ensure the system's normal operation.Finally,based on the research of this paper,the in-depth analysis and summary of the key technical points involved in this paper are carried out,and the direction of improvement of the existing problems and the work plan for the next step are proposed.