Study On Adaptive Problems In Sparse Coding Algorithms

The sparse coding algorithm was originally developed as a neural model for natural images, in order to simulate the visual processing of simple cells in the primary visual cortex (area V1) of mammals. The principle of sparse coding could be expressed that only a small number of simple cells are significantly activated for a given image signal. Equivalently, the idea of sparse coding is that a given simple cell is only rarely significantly active for a series of different image signals. Because it can successfully model the response properties of simple cells in the primary visual cortex of mammals, and can learn the characteristic basis functions from the statistics of the input images, the sparse coding algorithm is regarded as an effective neural coding method with an adaptive learning mechanism, which certainly has a promising feature. Actually, the sparse coding algorithm has been applied in image coding, image denoising, image retrieval, blind source separation, feature extraction and pattern recognition, etc. However, the sparse coding algorithm is still at a developing stage, for there remains a great deal that is still unknown about how the primary visual cortex works. Therefore, the theory and application study of sparse coding should be enhanced and improved further.In this dissertation, after a brief introduction to the neuro-biologic background of sparse coding, a simple mathematic description of sparse coding was given, and then the development history, the current research status and the shortcomings of sparse coding were introduced. In the following, some unresolved adaptive problems in the sparse coding algorithm were investigated at large, and three effective solutions were proposed to solve such problems. As a result, not only at the preprocessing stage, but also at the learning stage, the adaptive ability of sparse coding was improved, which was testified separately in the experiments of feature extraction, face recognition and source blind separation.The main works in this dissertation can be concluded as follows:First, an adaptive whitening/low-pass preprocessing method, based on the power spectrum properties of natural images, was proposed. In this study, the disadvantages of the fixed whitening filter used in the current sparse coding algorithm were firstly analyzed. To avoid such drawbacks, by deeply investigating the characteristics of the power spectrum of natural images, this dissertation constructed a flexible whitening filter, the flexibility of which was controlled by a variable whitening parameter. Because the optimal value of the whitening parameter was determined by the properties of the input images, the flexible whitening filter was adaptive to the data and thus was suitable for more images than the fixed one. Furthermore, considering that the whitening filter does not distinguish noise from useful signals, this dissertation combined a low-pass filter with the proposed adaptive whitening one. Then when the input images were preprocessed by the combined adaptive whitening/low-pass filter, not only the useful high frequency components, such as edges and lines, in the raw data were significantly enhanced, but also no significant noise was allowed to pass. Obviously, such a result was helpful to the sparse coding algorithm which was sensitive to the higher-order statistics of the input data. For example, in the experiment of feature extraction from natural images, it turned out that the whitening/low-pass preprocessing could pick up the convergence speed of sparse coding; in the experiment of face recognition by independent component analysis, it turned out that the whitening/low-pass preprocessing could improve the recognition rate (Note that the independent component analysis is equivalent to the sparse coding in the case of no noise and a square system, and therefore in this dissertation, the ICA method was regarded as a special case of sparse coding).Second, a sparse coding algorithm with Lorentz adaptive priors was proposed. In this study, after illustrating the Bayes exploration of sparse coding, it was suggested that because the coefficient priors were often uniformed and fixed during the learning, the current sparse coding algorithm was not only easily limited in applications, but also took departure from the adaptive learning mechanism which is popular in the visual system of mammals. To solve such problems, in this dissertation, it was first indicated that there was a need for the coefficient priors to be more flexible, and the flexibility could be obtained by adjusting the coefficient priors adaptively from the input data. Then it was deeply discussed that one of the generalized Cauchy distributions, also called Lorentz distribution, could be used to model the coefficient priors in the sparse coding framework. In the following, the specific learning rules for the sparse coding algorithm with Lorentz adaptive priors were inferred. The main difference of the new algorithm from the conventional one with uniformed and fixed priors was that at each iterative learning step, the new sparse coding algorithm needed to estimate the Lorentz scale parameter for every coefficient. Just because the coefficient priors can be adjusting adaptively from the input data, the coding efficiency of the sparse coding algorithm was improved, which was testified in the experiment of feature extraction from natural images.Third, a sparse coding algorithm with Pearson VII adaptive priors was proposed. In this study, the tail properties of the Lorentz density were analyzed first. It turned out that the tail index of the Lorentz density is equivalent and only equivalent to 2. That was to say that although the Lorentz prior model could be adaptive to the data for a variable scale parameter, the flexibility of which was still limited for the fixed tail index. To solve such a problem, in this dissertation, it was further suggested that one possibility to improve the flexibility was to introduce into the Lorentz density function a shape parameter which could describe the tail properties of the distribution function. Actually, such a generalized Cauchy distribution was the Pearson VII probability distribution. In this dissertation, the Lorentz distribution was regarded as a generalized Cauchy distribution with only one scale parameter; while the Pearson VII distribution was regarded as a generalized Cauchy distribution with a scale and a shape parameter both. For the sake of the added shape parameter, the flexibility of the Pearson VII distribution was stronger than that of the Lorentz one. Naturally, in the sparse coding framework, the Pearson VII adaptive prior model can capture the higher-order statistics from the input data to the better detail. Therefore, compared to the sparse coding algorithm with Lorentz adaptive priors, the sparse coding algorithm with Pearson VII adaptive priors could achieve a better coding efficiency, which was testified again in the experiment of feature extraction from natural images.Fourth, to apply the two generalized Cauchy prior models into the special sparse coding algorithm, that is ICA, to separate blind sources. The goal of this study was two-fold: One was to improve the separation performance of the conventional ICA method; the other was to further testify the effectiveness of these two prior models. It was well known that an appropriate selection of the nonlinear contrast function was the key for achieving successful separation. Therefore, if the nonlinear contrast function could be estimated adaptively from the input data, the separation performance could be enhanced for sure. In this dissertation, it was first suggested that the adaptive estimation of the nonlinear contrast function could be implemented by the way of adjusting the prior densities of the sources adaptively from the data; it was then discussed how to apply the two generalized Cauchy prior models, that were the Lorentz and the Pearson VII adaptive priors, into the ICA algorithm. In the simulation experiments, the two ICA algorithms with Lorentz and Pearson VII adaptive priors had proved more effective than the conventional ICA method with fixed priors; while the separation performance of the ICA algorithm with Pearson VII adaptive priors was better than that of the one with Lorentz adaptive priors.