Research Of Online Density Estimation Based On Incremental Gaussian Mixtures

Density estimation is a classical problem in statistical inferences and has been applied extensively to many fields in machine learning and data mining. As the rapid development of mobile phones and social networks, streaming big data has became the main subject of machine learning and data mining tasks. It has the properties of low latency, high throughput and continuously reliable running.Traditional algorithms for density estimation, whether parametric or non-parametric ones, do not scale to meet the requirements. Therefore, we need a new approach to re-alize the task of online density estimation for such streaming data. Parametric methods often make unrealistic assumptions on the density function to be estimated while non-parametric ones suffer from the unacceptable time complexity and space complexity. It is quite interesting that how to combine the strong points of both sides. We pro-pose a Local Adaptive and Incremental Gaussian Mixture(LAIM) for realizing online density estimation in answering this question and compare it with other state-of-the-art algorithms for density estimation, include both on-line and off-line ones.The main contributions of this paper includes:1. we first review some relevant concepts and models for density estimation;2. After analyzing the underlying probabilistic model of Self-Organizing Incremen-tal Neural Network(SOINN), a competitive neural network that can fulfill online unsupervised learning, we reveal the connection between SOINN and Gaussian Mixture Models(GMM):SOINN is inherently an incremental version of GMM, hence it is naturally a density estimation algorithm.3. To estimate locally complex density structure with fast convergence rate, we pro-pose a Local Adaptive and Incremental Gaussian Mixture(LAIM) for online den-sity estimation. It is able to allocate components incrementally to accommodate novel data points without affecting previously learned components and updating the model locally. The experiments on both artificial and real data sets show that our method outperforms the compared on-line counterpart and produces compa-rable results to the compared batch ones.