Nonlinear signal processing for digital communications using support vector machines and a new form of adaptive decision feedback equalizer

This dissertation addresses the general problem of correctly detecting information symbols in a communications channel having nonlinear dispersion. The solution of this problem is important to emerging transmission technology as well as existing voice band channels. There are four major contributions of this work.; The first contribution is the proposal and analysis of support vector machines (SVMs) as an equalizer for nonlinear channels. Results show that SVMs perform as well as neural networks on the nonlinear problems investigated. In addition, two methods are proposed to address the fact that intersymbol interference (ISI) generates input vectors having temporal correlation, whereas a standard SVM assumes independent input vectors. A simulation using a linear system shows that the second of the proposed methods performs equally to a conventional decision feedback equalizer for the studied problem.; The second contribution is a new method of performing multicategory classification called the M-ary SVM. The M-ary SVM represents each category in binary format, and to each bit of that representation is assigned a conventional SVM. This approach requires only [log₂(K)] SVMs, where K is the number of classes. Conventional methods often require on the order of K or K 2 classifiers. We consider an example of classification on an octaphase-shift-keying pattern space to illustrate main concepts.; The third contribution is a method of adding equality constraints to the conventional SVM to force its classifier boundary to pass through chosen points. Applications of this method often arise in problems having symmetry. We examine one such example where the M-ary SVM is used to classify symbols of a multiuser detection pattern space.; The fourth contribution is a variation of a conventional adaptive decision feedback equalizer which uses selection as opposed to nonlinear feedback to remove ISI. For the adaptive decision-selection equalizer (ADSE), past decisions choose different sets of filter and ISI removal coefficients of a linear model. The major advantage of the new method is improved performance on the studied nonlinear channel while retaining simplicity. A more robust version of the ADSE results when only the ISI removal constant is selected.