| In this thesis, we focus on digital signal processing (DSP) solutions that can help to combat various sources of nonlinearity in fiber-optic communication systems. Fiber Kerr nonlinearities are widely known to constitute a fundamental limit on the capacity of long-haul optical transmission as they restrict the maximum launch power into the fiber. This effectively limits the maximum optical signal-to-noise ratio (OSNR) that can be achieved in the receiver which in-turn puts a cap on the transmission reach. However, cost-effective DSP-based fiber nonlinearity mitigation schemes for long-haul transmission are yet to be deployed. By employing Volterra analysis of an optical channel comprising multiple spans of single mode fiber (SMF), we develop two solutions to improve the complexity-performance trade-off of Volterra-based nonlinear equalization (VNLE). First, we demonstrate that a significant portion of the VNLE filter coefficients are canceled out in the presence of a symmetric dispersion map. Additionally, we propose novel cascade structures for VNLE that are shown to provide substantial complexity reductions compared to the conventional VNLE with linear and nonlinear filters in parallel. Note that other sources of nonlinear distortions can also significantly impair system performance. Recent experiments on transmission of high baud-rate optical signal revealed that the back-to-back performance of these systems are highly degraded by pattern-dependent distortion (PDD) which cannot be effectively compensated by linear filtering or static look-up-tables. In order to address these impairments, we developed a family of sequential detection strategies based on hidden Markov modeling of PDD. The proposed solutions are highly configurable to suit the target complexity constraints. |
