Design And Implementation Of Cloud Protection Log System

The development of information technology is always accompanied by the openness,divergence and interactivity of data,so the exchange and sharing of data is also becoming possible.But in view of the objective existence of this feature,it also indirectly leads to the emergence of network security problems.In the field of network security attack and defense,the protection platform,as a defensive force,always has a congenital lag.Especially in recent years,with the rapid growth of enterprise business volume and the emergence of new technologies,data leakage,hacking and other network security accidents occur frequently.Traditional protection platform is gradually losing its role,the new protection platform solution is in urgent need of design.The network security equipment in the protection platform will not only intercept threats,but also generate a large number of logs in operation.Through the integration and analysis of the data in the log,not only can the target and source of the threat be found,but more importantly,the threat perception can be realized by using the algorithm to prevent the threat from being invaded again.However,with the advent of the era of big data,not only the intrusion protection pressure of the protection platform significantly increases,the scale of log also presents exponential growth.At the same time,the distribution,heterogeneous diversity and dynamic variability of the log also pose challenges to the design of the log system under the protection platform.Traditional log system is mainly based on the single-node server,and its performance is very limited in computing,I/O transmission,storage and other aspects.At present,log data is gradually emerging new features such as mass,decentralization and isomerization,and the traditional log system has been unable to meet the needs of large-scale log collection,processing and storage.In view of the dilemmas of the traditional protection platform and the log system under the background of the information and big data,this paper briefly discusses the construction of the SaaS cloud protection platform and On-Premises protection platform and expounds the construction and design of the log system of the SaaS cloud protection platform,through the research of the theory and achievement of the big data,distributed and cloud computing.1)Design of network security protection platform.By analyzing the working principle and defects of traditional protection platform,SaaS cloud protection platform and on-premises protection platform are designed.SaaS cloud protection platform relies on big data and cloud computing technology.Compared with traditional protection platform,SaaS cloud protection platform has great advantages in system computing performance and later maintenance cost,etc.It is currently a widely used protection scheme.On-premises protection platform,as the evolution of local protection technology,is more a technical supplement to the mainstream SaaS cloud protection platform,and is more used in scenes with relatively high security level.2)Construction and design of cloud protection log system.Functional requirements analysis divides the design of the system into the following three functional modules:log collection,log processing and data storage.In the log collection module,the open source log collection system Flume is adopted for distributed collection and large-scale aggregation of logs.To solve the efficient real-time collection of massive logs,the secondary transmission structure of Flume Agent and Flume Collector is designed.In view of the rate mismatch between log collection and log processing modules,message queue Kafka is adopted between modules,which is responsible for the caching of logs and coupling between modules.The log processing module is divided into two units:real-time computation and offline computation.Real-time computing rely on Storm to achieve the real time cleaning and log data statistics,and offline calculation is based on the research of the Map/Reduce and the related algorithm,and realized the log data mining and security audit,etc.The data storage module is divided into log storage and data report storage.The log storage design adopts the hybrid storage scheme of HBase and HDFS.HDFS serves as the underlying storage structure of massive logs,while HBase realizes the external interface at the business level.Data report storage adopts the classic design of MySQL+Redis background management system,which realizes the storage and display of log processing results.3)research on related algorithms of cloud protection log system.The parallel clustering analysis of Map/Reduce and the realization of HBase performance optimization are mainly studied in the related algorithms of the system.Among them,the data mining algorithm of off-line computing mainly adopts the unsupervised learning k-means algorithm,which is combined with the Map/Reduce computing model to realize the parallelization clustering analysis of massive logs,and verifies the superiority of Map/Reduce cluster parallelization processing in the large-scale log data mining compared with the traditional single-node processing.HBase performance optimization relies on the research on the storage principle of HBase and the key analysis of "hot spot"phenomenon in HBase.By using the prepartition of HRegion and the random hash optimization method of RowKey,the unbalanced problem of data access in HBase is effectively solved.