Theoretical and experimental analysis of the generalization ability of some statistical learning algorithms

This thesis considers problems arising in the area of non-parametric regression and classification when using combinations of learning machines.; New measures of complexities of convex combinations of classifiers from Generalization bounds taking into account these complexities are derived. The approaches taken here for the exploration of these problems are based upon statistical learning theory, empirical process theory, approximation theory and regularization theory. Previous results are described and compared theoretically and experimentally with new results presented here. The generalization bounds derived and these new measures of complexity of convex combinations of classifiers are shown to be useful for model selection.; Classifiers or regressors can be combined using component-based hierarchies of kernel machines. Interest in applying hierarchies in learning problems has grown recently. Some new theoretical results on the generalization performance of component-based hierarchies are presented here. The generalization bounds derived here depend on the empirical margin and the sum of the weights of the component kernel classifiers derived at different levels of the hierarchy.; Lastly, generalization bounds are shown for convex combinations of classifiers from random classes, i.e. function classes depending on the training set and invariant under permutation of the set and possessing the increasing property. Examples of convex combinations of classifiers from random classes are classifiers generated by Support Vector Machines, boosting with discretized decision stumps, radial basis function networks or some hierarchies of kernel machines. The approach taken for deriving the generalization bounds introduces the equivalent to Vapnik's “optimistic” inequality for random classes of sets. Some measures of complexities, such as sparsity of the weights and cluster variance for the convex combinations of classifiers, are considered as well. Whenever possible, these bounds are compared with previous bounds.