| Clustering analysis is an important technology of data mining,which has extensive applications in various domains,including web search,text processing,medical diagnosis.The aim of clustering is to group a set of objects into clusters so that the objects in the same cluster have high similarity but are very dissimilar with objects in other clusters.The traditional clustering methods can be roughly divided into partition-based clustering-hierarchical clustering,grid-based clustering,density-based clustering,model-based clustering etc.Non-uniform data is a common data set in real life such as medical diagnosis and forest disaster monitoring.Compared with Imbalanced data,non-uniform data can describe such data which are large difference in sample size and density among clusters more accurately.Up to now,Clustering analysis of such data is a difficult task in the field of data mining.Researchers have done some related researches on this difficult problem,but there are still some problems.The relatively typical ones are as follows:(1)K-means-type algorithms have "uniform effect" when clustering non-uniform data.(2)Currently,non-uniform data clustering algorithms are mostly defined in the full space and do not consider subspace scenarios.(3)How to cluster large-scale non-uniform data efficiency.To settle the problems listed,we have done some related research.The majority of our contributions can be summarized as follow:(1)A non-uniform clustering algorithm based on coefficient of variation is proposed to solve the problem of the "uniform effect".Instead of the squared error,a new measure of dispersion for non-uniform data was proposed relied on the coefficient of variation.The clustering objective optimization function was defined using a new non-uniform data dissimilarity formula,which was proposed based on the coefficient of variation.According to the local optimization method,the clustering algorithm process was given.The experimental results show that the algorithm improves the clustering accuracy on non-uniform data.(2)A soft subspace clustering algorithm for non-uniform data is proposed.Firstly,the subspace clustering model is introduced.Then a soft subspace Gaussian mixture distribution model for describing non-uniform data clusters is proposed.This model allows datasets to contain both clusters with different densities and sizes.Secondly,non-uniform data clustering is deduced,meanwhile the EM-type clustering algorithm to optimize the function is defined.Finally,Application analysis is conducted on medical diagnosis and other real data and the experimental results show that the new algorithm has higher clustering accuracy.(3)A non-uniform data clustering algorithm is realized in the Spark platform to cluster large-scale non-uniform data.Firstly,the algorithm based on the non-uniform data clustering algorithm proposed in this paper is designed to adapt to the conditions required by the parallel distributed platform.Then the algorithm is implemented in the Spark platform and tested with experimentally. |
