Design And Implementation Of Anomaly Detection System Based On Muti-Source Heterogeneous Data Fusion

With the rapid development of the Internet of Things,big data technologies and applications,the operation monitoring system can more comprehensively perceive the operation status of each part and level of the system.The operation monitoring data presents the characteristics of multiple sources,and its richness and quantity are greatly improved.At the same time,these data are often heterogeneous,that is,the data formats and structures of different data sources are different.In this context,system anomaly detection based on multi-source heterogeneous data has become one of the hot spots in the industry.This paper aims to design and implement an anomaly detection system based on multi-source heterogeneous data fusion.The system can realize the collection,aggregation and storage of multi-source heterogeneous data,and support the anomaly detection function based on multi-source heterogeneous data.The system supports the collection of multi-source heterogeneous data and the synchronous import of data in the existing database through the lightweight collector beat,and then realizes the data engine based on ElasticSearch,supports data persistent storage and provides data query management;At the same time,two anomaly detection services are implemented based on Flink:1)Anomaly detection based on threshold;2)Model based anomaly detection.This paper presents an anomaly detection method based on multi-source heterogeneous subsequence multi view fusion(Anomaly Detection method based on Multi-source heterogeneous Subsequence multi view Fusion,MSVFAD).Firstly,an improved shape based subsequence embedding method is proposed,which convolutes the subsequence into a vector,selects a new vector base based on the shape again,rotates and reduces the dimension to form a two-dimensional shape based subsequence feature space,and completes the feature extraction of the subsequence;Then,based on the improved D-S(Dempster/Shafer)evidence theory,a unified feature fusion matrix is constructed,which is composed of the diagonal block matrix constructed by the similarity matrix of each source to form a complete feature fusion matrix,so as to further solve the problem of difficult fusion of multiple sources;Finally,an anomaly scoring method based on property attribute fusion is proposed by using multi view fuzzy clustering,which solves the difficulty of multi view anomaly detection,and introduces the importance factor of data source for adjustment for the first time.A series of experiments show that msvf-ad has higher accuracy than baseline algorithms such as MSCAN and DBN.First of all,this paper introduces the research background,current situation and the research content of this subject;Then the anomaly detection and related data fusion technologies are introduced;Firstly,this paper introduces the research background,current situation and the research content of this subject;Then the anomaly detection and related data fusion technologies are introduced;After that the data fusion anomaly detection system and solution are investigated,and then the requirement of anomaly detection system based on multi-source heterogeneous data fusion is analyzed.Later,an anomaly detection method based on multi-source heterogeneous subsequence multi view fusion is proposed,and the superiority of the algorithm is proved by a series of experiments;Subsequently it introduces the whole system and the design of each module;Finally,the implementation of the system is explained,and the effectiveness of the system is verified by a series of tests.