Research On Key Techniques Of Anomaly Detection For Big Data Platform Based On Dynamical Rule Base

With the increase of Web applications,the big data platform architecture is also becoming more and more complex.And,Web applications are directly related to the interests of individuals and businesses,frequent attacks on Web applications lead to a huge challenge of the security for big data platform.An effective solution to ensure the security for big data platform is using anomaly detection technology.However,there are some problems with existing methods.Firstly,the massive and timeliness of Web traffic may causes that the existing anomaly detection technology failed to detect anomalies in a short the.Secondly,the existing anomaly detection techniques are mostly concentrated in a period of time abnormal detection,but its research on real-time positioning for anomaly is short,which result in the follow-up processing not timely;Thirdly,experts have to spend lots of energy and time to update and maintain the rule base of anomaly;Fourthly,domestic and foreign scholars pay a large attention to the development of the security technology for big data platform,who carried out lots of work but which really achieved practical ware not much.Therefore,how to quickly and efficiently detect anomaly of big data platfonn is of great significance for improve the security of big data platform.The subject comes from the major science and technology projects in Fujian Province "Research and Development of Key Techniques of Real-time Anomaly Detection and Analysis System for Big Data Platfonn",which had analyzed the currently security problems of big dada platform.The subject had studied the Anomaly Detection for big data platform based on Dynamical Rule Base(AD_DRB).(1)The Dynamic Rule Base construction method based on Maximum Frequent Patterns(DRB_MFP)was proposed.Firstly,the Web Access Sequences Database(WASD)was extracted to be converted into Dynamical Web Access Sequences Database(DWASD).Secondly,the maximum frequent pattern was mined from DWASD on Spark platform.Thirdly,the dynamic rule base was constructcd according to maximum frequent patterns.(2)The Distributed real-time anomaly location method based on Web session flow was proposed.Firstly,the session sequence was aligned with the rule sequence by similarity measure of hybrid biological gene sequence alignment algorithm.Secondly,the anomaly data was accurately and efficiently positioned by detecting real-time Web session flow,with the support of Spark Streaming technology.(3)The anomaly detection and analysis system prototype for big data platform was designed,which was expected to provide some reference for the industry.By having experimented with real mass data sets,AD_DRB could detects anomaly quickly with an anomaly precision rate of more than 85%and an anomaly recall rate of more than 80%.The experimental results show that,AD_DRB had a good time performance and anomaly detection sensitivity,which was capable of being effectively applied in improving the security of big data platform.