The Cluster Analysis And Anomaly Detection Based On Density Core And Local Resultant Force

The development of science and technology,especially the rapid rise of intelligent mobile interconnection and the Internet of Things technology,has a great impact on the way of data generation as well as the dimensions,the sizes and the types of data,which increases the complexity of data and makes it more difficult to label data.Therefore,how to solve problems which are related to pattern recognition in samples with unknown class labels has become the priority of unsupervised learning applications.Clustering and anomaly detection are two widely used methods in unsupervised learning.In China,the idea of clustering has a long history and the thought that birds of a feather flock together come into being very early.In recent years,the concept of density peak has been widely used in the field of clustering learning.Relevant methods combining density peak and clustering algorithms have achieved good results in image processing and pattern recognition.However,when dealing with complex data with arbitrary shapes,high dimensions,multi-density hierarchies and noises through a single density peak,the cluster information cannot be depicted completely,which has a negative impact on clustering results.In this thesis,the concept of density core is introduced into clustering analysis.Through density core,the attributes of clusters can be represented and described effectively,which solves the problem that existing clustering methods based on the single density peak cannot be applied to data sets containing complex shaped clusters.Clustering analysis is applied in the selection of general patterns,while anomaly detection which represents valid,interesting,and potentially valuable patterns is often more magnificent than detecting normal observations.Because of its simplicity,rapidity and effectiveness,anomaly detection based on distance and density has been widely used in recent years.However,parameter-sensitive defects often lead to instability of detection performance of these methods.Inspired by gravity,this paper presents a local resultant rate of change model,which is introduced into the detection of outliers and boundary points.Insensitive to input parameters,the detection method based on local resultant force change rate can effectively detect outliers and boundary points.In this thesis,we apply the density core in clustering analysis and the local resultant force in outlier detection.The main contributions and innovations are as follows:(1)In this thesis,we propose a clustering method based on density core and dynamic scanning radius(DCNaN)which combines the concept of natural neighborhood and improves traditional density core acquisition methods through dynamic scanning radius r without manual parameter input.The process of obtaining Mean-Shift and the density core based on dynamic scanning radius r can automatically adapt to multi-density level data sets and solve the problem that existing clustering methods can not deal with data sets with complex shape clusters and multi-density levels.(2)In this thesis,we propose a Relative Density-core-based Clustering Algorithm with Natural Neighbor(RDcore)which automatically acquires the K value through natural neighbors.A method based on natural neighbors NaNRNKD(Relative K-nearest Neighbor Kernel Density Measurement Based on the Natural Neighbor)is proposed.According to NaNRNKD,a neighbor can quickly and accurately acquire data points belonging to the density core,which solves high time complexity of the acquisition of density core through mean shift.(3)In this thesis,a new internal index based on the density core for clustering validation(DCVI)is proposed.Through density core,DCVI transforms the internal evaluation index of all data into the internal evaluation index of the density core and proposes the internal evaluation of clustering according to the compactness within clusters and the separation between clusters.Indicators can effectively avoid the impact of noise,complex shape and overlap on Evaluation indicators.Combining the Minimum Spanning Tree(MST)clustering method,this thesis p resents a DCVI-MST clustering method which can accurately and quickly evaluate the number of clusters and give effective clustering results.At the same time,DCVI can be effectively applied to hierarchical partition-based,density-based and K-means clustering methods in the selection of the optimal number of clusters.(4)In this thesis,an outlier detection method based on local resultant change rate(LGOD)is proposed.In order to eliminate the shortcomings of traditional outlier detection methods that are sensitive to parameters and in need of artificial parameters,LGOD uses local resultant change rate to obtain the ranking of outliers,boundary points and interior point change rates,and applies hierarchical partition method in the selection of outliers.(5)In this thesis,we propose a boundary point detection method based on local resultant force change rate(LGBD)which obtains more accurate boundary points in data sets through the local resultant force rate of change and proportional parameters.Compared with other methods,LGBD can better reflect the distribution characteristics of data.Through experimental comparison between artificial data sets and real data sets,related theories and methods proposed in this thesis are obviously superior to existing related methods and theories.Through analysis and researches of the basic theory and related algorithms in the fields of clustering and anomaly detection,we hope that we can enrich existing unsupervised learning methods.Furthermore,we also provide corresponding solutions to related problems and bottlenecks in fields of clustering analysis and anomaly detection.