A Novel Approach For Intrinsic Dimension Estimation Based On Maximum Likelihood Estimation

Since the 20th century 60's,the rapid development of computer technology provides a powerful tool for dealing with complex data, which also makes the high-dimensional data analysis techniques have vigorous developing. The intrinsic dimension estimation of high-dimensional data, is important in the field of high-dimensional data processing. It has a very high importance and urgency for finding the intrinsic dimension. Intrinsic dimension is an unknown quantity which needs to be estimated for dimension reduction algorithms, if we can quite accurately find the intrinsic dimension of high-dimensional data, there was not doubt that the right dimension estimation has an important guiding significance for dimension reduction algorithms. The accurate intrinsic dimension estimation also profit to select an appropriate neighborhood size in data processing for avoiding dimensionality curse.This paper proposes a new approach for intrinsic dimension estimation based on MLE (Maximum Likelihood Estimation). Generally, neighbor relationship can adequately reflect the local geometric characteristics of the data. MLE method constructed the likelihood function between the distances of close neighbors, to get the maximum likelihood estimate intrinsic dimension. The traditional MLE algorithms has two shortage, one is that it just simply making an average for the intrinsic dimension estimation of different sample points in the same neighborhood, which will be subjected interference because of Singular value, the other is that it uses Euclidean distance to option the k neighbors, which likely appeared layers phenomenon. In order to solve the shortage, we use NS (neighborhood smoothing) algorithms instead of mean algorithms, and use geodesic distance instead of Euclidean distance to find the real K-neighbor points when we select K neighbors.We make the experiment on simulation data sets and real data sets, the results showed that our method is effective, it can find more reasonable intrinsic dimension.