Generalization of widely linear filtering concepts for equalization and interference suppression in PAM/QAM systems

In PAM/QAM transmission systems, standard minimum mean square error (MMSE) linear (LE) or decision feedback equalization (DFE) methods are often used to suppress the detrimental effects inter-symbol and/or co-channel interference (ISI/CCI) effects of the channel. The main aim of this thesis is to show the benefits of widely linear (WL) filtering for equalization applications in wireless systems. For this purpose, we introduced generalized widely linear MMSE/MMSE-DFE equalization structures for both PAM systems that use real constellations and QAM (that use complex modulation alphabets) with multiple antennas and multiple co-channel interferers. In the proposed implementation, the WL receiver, unlike conventional methods, first separates the in-phase (I) and quadrature (Q) parts of the complex baseband received signal and jointly filters the two branches for signal detection. We derived the filter settings for both infinite and finite length implementation and analyzed the receiver performance in various channel conditions. This thesis has three main contributions. First, we analyze the advantage of WL filtering for PAM signaling in a white noise channel. When the ISI channel response is complex, we show that WL LE/DFE receivers outperform conventional methods both in complexity and performance. We obtained closed form mean square error (MSE) expressions that clarified the MSE advantage, and lower noise enhancement properties of WL receivers. Second, we show that WL receivers outperform conventional receivers in the presence of PAM-type interference. For both PAM and QAM signaling, assuming that the WL-DFE feedback path is error free, we show that, a receiver with N antennas can reject up to 2N - 1 PAM interferers or any combination of M1 PAM and M 2 QAM interferers satisfying the constraint: M 1 + 2M2 < 2N. This is significant gain compared to a conventional receiver whose interference cancellation (IC) capability is limited to N - 1 interferers at most. Third, we analyze the symbol error rate (SER) performance and diversity-IC trade-off for the proposed receivers in a flat Rayleigh fading channel. After deriving an upper bound (UB) to the SER and certain approximations to the UB, we used the SER expressions to characterize the trade-off between diversity and interference cancellation. A comparison between conventional and WL methods showed that the additional dimensions created by I-Q split enhances the diversity gain in addition to providing IC advantage.