Model selection using statistical learning theory

In recent years, there has been an explosive growth of methods for predictive learning in the fields of engineering, computer science and statistics. The goal of predictive learning is to estimate (or learn) dependencies from (known) training data, in order to accurately predict (unknown) future data originating from the same (unknown) distribution. Model selection is the task of choosing a model of optimal complexity for the given data. It is one of the major issues in predictive learning with finite samples. This research work is focused on the fundamental issues as well as practical applications of model selection. Our approach is based on Vapnik's statistical learning theory (SLT), a recent important theoretic framework for learning with finite samples. Important contributions of this research include:; VC-based model selection. We have studied the practical feasibility of VC-based model selection (complexity control). We have compared VC generalization bounds for model selection for linear and penalized linear estimators with classical model selection methods. We have also proposed a new method for wavelet signal denoising based on VC-theory.; Measuring the VC-dimension. We have proposed an optimized experimental design technique for accurate estimation of the VC-dimension, which is the measure of model complexity in VC-theory. We have estimated the VC-dimension for linear and penalized linear estimators using the proposed optimized procedure. We have also developed a method for estimating VC-dimension for Constrained Topological Mapping (CTM) networks.; Support Vector Machine (SVM). SVM is the new type of universal learning machines based on VC-theory. We have proposed an important extension to SVM: the Multiresolution Support Vector Machine methods. We have also proposed the VC-based model selection for SVM. on for SVM.