| This thesis provides signal-processing and constellation shaping algorithms to combat equalizer-enhanced phase-noise and nonlinearity-induced interference in long-haul fiber-optic communication systems.;In the first part of this work, an iterative phase-estimation algorithm is designed that combines expectation-maximization (EM) with a soft-input soft-output error control decoder. At 280 Gbit/s, this system doubles the optical system reach, and generally enhances phase-noise tolerance. Among three different 16-point constellations, the 4-4-4-4 ring constellation was found to have the best performance. The laser linewidth tolerance gain is improved further by reducing the code rate.;In the second part of this work, this code-aided (CA) EM approach is applied to long-haul systems dominated by nonlinearity-induced interference. Based on laboratory measurements, a second-order autoregressive phase-noise model is proposed, and used to modify the EM regularizer term. Simulation and experimental results show that in a dual-polarization wavelength-division-multiplexed 16 QAM system, launch-power tolerance can be increased by 1.5 dB, and the optical signal-to-noise ratio requirement can be relaxed by 0.3 dB to achieve the same error ratio. The complexity of the CAEM algorithm was found to be about 1/4 of the complexity of digital back-propagation.;In the third part of this work, a low complexity probabilistic constellation shaping scheme is proposed to increase the system tolerance to nonlinearity impairments. A theoretical analysis predicts that, with 16 and 64-point constellations, reach improvements of 7% and 10% respectively can be achieved at a mutual information of 3.2 bit/symbol/polarization. Such probabilistic constellation shaping requires minimal added complexity. |
