Research And Implementation Of Security Situation Awareness System Based On Hadoop

In the big data environment,network security events are constantly emerging,and the diversification and refinement of hacker technology has brought huge challenges to network security.Traditional firewall protection,data encryption and other security measures have been unable to meet the needs of network security protection,and security situation awareness The application of the system is imperative.This paper designs a secure situational awareness system based on the Hadoop architecture.According to the requirements of the security situational awareness system on the clustering algorithm,the traditional K-Means algorithm is improved,and the parallelization is implemented based on the MapReduce programming model.The parallel clustering algorithm and the traditional attack library matching are combined to realize log security analysis,and a visual platform is built for real-time network security monitoring,alarming and analysis.The main work of the paper is as follows:1.This paper proposes to improve the K-Means clustering algorithm,optimize the algorithm flow,overcome the random defects of the traditional K-Means algorithm when looking for the cluster center,and the defects of the clustering effect interfered by isolated points and noise points.MapReduce parallelization research was conducted to adapt it to the requirements of big data processing,greatly reduce the algorithm time,improve parallelization performance and attack detection rate accuracy.When analyzing logs,the cluster-based anomaly detection method and the rule-based anomaly detection method are combined.When analyzing the log,the cluster-based anomaly detection method and the rule-based anomaly detection method are combined,and the improved K-Means algorithm is used to analyze the log data to identify the identified threat data and send it to the local attack database The second match improves the accuracy of threat identification.2.Design and implement a security situational awareness system,build the system and configure the environment,including setting up 7 virtual machines,IP configuration,installing and configuring Java,installing and configuring ssh and Hadoop,etc.The functions implemented by the system include: collecting and persistently storing data to meet the requirements of large data storage of about 1500 TB / year;preprocessing the stored real-time data stream,including cleaning,filtering,and data standardization;Identify threat information in real-time data streams,real-time statistics of existing threats,and provide early warning of external threats and vulnerabilities for unknown threats;display the results of security data analysis in multiple scenarios with multiple types of charts,including security postures,System security warning,system security warning,system risk quantification and prediction,system asset security,system user portrait,system attack portrait,etc.,and provide platform management functions for users.The security situation awareness system designed in this paper achieves real-time detection of network security status,including system security and asset security.Generate real-time alerts on existing security threats,comprehensively evaluate system users and attack behaviors,determine the risk level,and predict future security events.The system has been put into practical application and has a high attack detection rate.