The main topic of this work is the application of advanced Digital Signal Processing
(DSP) techniques to high data-rate optical links. The first part of the thesis is devoted to direct-detection systems, with a specific focus on data-center (DC) communications. These links, until recently, used very simple On-Off Keying modulation. However, the strong push towards higher data-rates introduced the use of some DSP algorithms in the DC. First, this thesis introduces a novel architecture for 400-Gbit/s Intra-DC (shorter than 2 km) applications. This architecture uses each fiber pair in both directions, doubling the per-laser and per-fiber capacity. Crosstalk is reduced by using a short feed-forward adaptive equalizer and a small frequency shift between lasers transmitting over the same fiber. This system, after an initial theoretical evaluation, was successfully tested with an experiment. Then, Intra-DC links were analyzed, i.e. links interconnecting different DCs in the same region. The target distance was 80 km. For this purpose, Single Side-Band (SSB) modulation was compared with Intensity Modulation/Direct Detection schemes over dispersion-uncompensated links with single-photodiode receivers. Using, as an application example, the DMT modulation format, the comparison was carried over different span lengths. It was found that, for distances longer than 10 km, SSB has a significant advantage over IM/DD systems.
The second part of the thesis focuses on coherent systems. In particular, it studied constellation shaping techniques, which have been recently proposed for optical communications. To perform a fair analysis, shaped constellations were compared, at the same net data rates, with standard lower-cardinality QAM constellations. It was found that, over a linear channel, the gain of constellation shaping is approximately 25%. Afterwards, this thesis discusses the non-linear properties of the optical channel, which are known to be enhanced by constellation shaping. In particular, it focuses on the generation of Non-Linear Phase Noise (NLPN) at different system conditions, using both standard QAM and shaped constellations. It was found that, in some scenarios, such as low symbol rates or low dispersion fibers, NLPN significantly degrades the performance of shaped constellations. Then, this thesis proposes some compensation techniques, which are able to partially mitigate the effect of NLPN in those conditions.