Properly specified functional mappings and support vector learning machines

Feature extraction is a fundamental problem in pattern recognition and image processing. It is a process of mapping a set of original measurements into a parsimonious set of sufficient statistics. This involves finding a best subspace that preserves class separability in the lowest possible dimensional subspace. I will show that finding the best subspace for the classification problem requires solving a fundamental problem in machine learning known as the bias and variance dilemma. The dilemma involves the system representation problem for functional mappings. Practical machine learning calls for data-encoded system features that can be hard-wired into functional mappings with finite amounts of data. I will show that the right functional bias for a particular problem involves an appropriate set of data-encoded system features based on a complete and sufficient set of eigenstructures.;Efficient functional mappings transform masses of data into parameter sets that encode relevant system features. Principal component and eigenvector directions are essential system features for the classification problem. I will show how eigenstructures are an inherent part of the SVM functional basis in that it encodes the geometric features of a separating hyperplane. SVM architectures based on insufficient eigenstructures are shown to have insufficient learning capacity.;Linear separating hyperplanes are optimal decision boundaries for normally distributed data sets that have the same covariance matrix. This is known as the linearly separable binary classification problem. I will identify a functional mapping for the Bayes separating hyperplane. It will be shown that the estimator encodes the right bias, and thus minimizes the Bayes risk.;SVM feature estimates are based on the inverse of an autocorrelation matrix that enters into the Wolfe dual solution. SVM autocorrelation matrices with insufficient rank are singular, and so are non-invertible. The SVM problem is ill-posed for the feature vectors considered in this dissertation because the feature vector dimension is less than the number of training points. I will develop a regularization technique for linear SVM that addresses these eigenstructure deficiencies. The regularization method developed in this dissertation is denoted by L2 SVM.;SVM returns large numbers of features for overlapping distributions. A feature extraction method is optimal if the information is minimally degraded. I will develop a principal component feature extraction method for both linear and nonlinear SVM. The SVM solution is equivalent to a weighted version of principal components analysis (PCA), where the data and SVM weight vectors are projected onto eigenvectors associated with the signal and noise correlation matrix. These results are used to develop an expression for the SVM decision function based on a smaller set of transformed support vectors, regardless of whether the classification problem is linearly or nonlinearly separable. The PCA compression technique offers an effective method for selectively tuning, in a refined manner, the amount of support vectors (SV) used to construct SVM decision boundaries. SV-PCA features provide core components for matched filter banks for both binary and m-ary classification problems. For the detection problem, the decision framework is the Bayes system.;For any machine learning classification method, the benchmark is Bayes method that establishes an optimal classification framework. The Bayes likelihood ratio defines an optimal test statistic that is the minimum cost or minimum probability of error for making a decision. I will show that the statistic for the L2 SVM Hypothesis Test is equivalent to the statistic for the Likelihood Ratio Test.