| An adaptive algorithm can be used to find a least-square-error solution for linearly estimating a random process given observations of a related random process. Such an algorithm can iteratively update a weight vector, an approximation of the optimal Wiener solution, as input data is presented in a streaming way. The Least Mean Square (LMS) algorithm is one of the most popular adaptive algorithms because it is simple and robust, as it is used in many commercial applications of significance. However, the performance of LMS may vary greatly when the autocorrelation matrix of its input has a high eigenvalue spread, so there is a need to be able to predict its performance in practice. The LMS/Newton algorithm is an ideal variation of the LMS algorithm that is immune to the eigenvalue spread problem, and it is commonly used as a theoretical benchmark for adaptive algorithms. In this thesis, we study the performance of LMS relative to LMS/Newton. The analysis is done for stationary and nonstationary signal statistics. In the stationary case, transient behavior results when the adaptive weight vector starts from initial conditions and proceeds toward the Wiener solution, and in steady-state, the adaptive weight vector hovers randomly about the Wiener solution. In the nonstationary case, the Wiener solution varies randomly, and the adapting weight vector is tracking a moving target. In both cases, simple expressions are found for the performance of LMS relative to LMS/Newton. When the input autocorrelation matrix is Toeplitz (as would be the case with an adaptive transversal filter), these expressions are translated to the frequency domain. Our results imply that, if the input power spectrum is similar to the spectrum of the Wiener solution (e.g. a low-pass input and a low-pass Wiener filter), the transient performance of LMS is better than that of LMS/Newton, in spite of a high a eigenvalue spread. If the above spectra are dissimilar (e.g. a low-pass input and a band-pass Wiener filter), LMS/Newton would be faster than LMS. In the nonstationary case, if the input spectrum is similar to the spectrum of the time variations of the Wiener solution, LMS tracks better than LMS/Newton in steady-state. Otherwise, the tracking performance of LMS/Newton would be superior to that of LMS. |
