Fast Dynamic Control of Optical Data Center Networks Based on Nanoseconds WDM Photonics Integrated Switches

Full text

OECC/PSC 2019
©IEICE
Fast Dynamic Control of Optical Data Center
Networks Based on Nanoseconds WDM
Photonics Integrated Switches
Xuwei Xue, Kristif Prifti, Bitao Pan, Fulong Yan, Xiaotao Guo, Nicola Calabretta
IPI-ECO Research Institute, Eindhoven University of Technology, Eindhoven, the Netherlands
x.xue.1@tue.nl
Abstract: We demonstrate a fast and dynamic control of photonic switches based DCN by a distributed
optical-flow control. Results show error-free 10 Gb/s switching with < 2dB penalty, no packet-loss at
switch and 260.66ns end-to-end latency.
Keywords: Data Center Network and Subsystem, Optoelectronic and all-optical switches.
INTRODUCTION
With the emerging of cloud computing, artificial intelligence, and the incoming of 5G mobile communications, the
traffic communications inside the data centers (DC) are imposed stringent requirements in terms of low latency, high
capacity, and high cost and power-efficiency [1]. With the aim to satisfy the scalable growth in both network traffic
volume and connected endpoints while decreasing the cost and the energy consumption, transparent optical DC networks
(DCNs) based on fast optical switches have been considered, featuring the data rate and format transparency and
eliminating the power consuming O/E/O conversions [2].
An optical DCN architecture based on distributed nanoseconds photonic integrated switches (PIC) has been proposed
and numerically investigated in [3]. The statistical multiplexing and the high-throughput provided by the photonics
integrated switch allows for efficient resource utilization, low latency, and high capacity and connectivity. However,
despite the nanoseconds reconfiguration time of the photonic switch, a fast dynamic network controlling mechanism that
enables the fully exploitation of the statistical multiplexing and improves the network throughput is essential. Moreover,
the lack of optical buffer at the optical switch results in large packets loss as the traffic load increases. Recently, a dynamic
network control mechanism with packets contention resolution protocol named optical Flow Control has been proposed
and numerically assessed to control the DCN and prevent the packets loss by packets retransmission [4]. However, the
fast dynamic control of the DCN based on the optical Flow Control protocol has not been experimentally implemented
and assessed to validate the network performance and to achieve the fast control of the WDM photonic switches.
In this work, FPGA is utilized to implement the ToRs and the switch controllers with the Flow Control protocol of the
photonics switch based DCN to achieve the fast dynamic control of the network and thus improving the utilization of
statistical multiplexing and high network throughput. Moreover, optical Flow Control protocol is deployed between the
FPGA-based controllers and ToRs to enhance the fast control mechanism preventing packets loss due to the lack of optical
buffers. Experimental results confirm that the fast dynamic control operation between ToRs and optical switches with no
packet loss at the photonics switch, and validate the switching operation in space, wavelength, and time domain of the
photonics integrated switch. Deploying the fast control, the photonics switch based DCN can error-free switch the 10
Gb/s data packets with <2dB penalty and 260.66 ns ToR-to-ToR latency. .
FAST DYNAMIC CONTROL SYSTEM
The architecture of the proposed DCN is shown in Fig. 1(a) and numerically investigated in [5]. N racks are grouped
into one cluster and there are N clusters in the proposed DCN. The intra-cluster switch (IS) and inter-cluster switch (ES)
are used to forward the intra cluster and inter cluster traffic, respectively. The i-th ToR in each cluster are interconnected
by the i-th ES (1≤i≤N). Every photonics switch has a corresponding FPGA-based switch controller to reconfigure the fast
switch and then forward the traffic packets. Each FPGA-based ToR generates data traffic consisting of the optical
Fig. 1: (a) DCN architecture employing the nanoseconds and buffer-less photonics integrated switches. (b) Schematic of the photonics
integrated switch and switch controller. (c) The fabricated photonics integrated switch chip.

OECC/PSC 2019
©IEICE
payloads and optical labels carrying the switch port destination information. One copy of the payloads is stored in the
electrical buffer of the ToR and it will be released or retransmitted based on the Flow Control feedback from the switch
controller. The optical labels are processed by the switch controller, which extracts the destination information, checks
possible contentions with other packets with the same destination, and thus sets the SOA gates inside the photonics
integrated switch to forward the optical payloads. Distributed optical Flow Control is deployed between the ToRs and the
IS/ES switch controllers to solve the packets contention, thus preventing packets loss at the photonics switch. After the
packets contention check by the switch controller, a positive acknowledgement (ACK, successful forwarding) is sent back
to the ToR, and the optical payload is released from the ToR buffer. In response to a negative acknowledgement (NACK,
packet dropped due to the contention), the ToR retransmits the optical payload. Note that given the proposed DCN
architecture, the implemented optical Flow Control operates in a fully distributed way, which enhances the scalability and
decreases the complexity of the DCN control as well as average latency.
The buffer-less photonics integrated switch is schematically shown in Fig. 1(b). The arrayed waveguide gratings
(AWG) groups WDM wavelengths coming from different ToRs and each respective optical module consists of a 1: N
splitter to broadcast the WDM channels to the N wavelength selective switches (WSS). The outputs of the N WSSs are
connected to the respective N output ports. Each WSS can select one wavelength channel and forward the channel to the
output port according to the switching control signals. Turning on/off the N SOAs determines which wavelength channel
is forwarded to the output or is blocked. It is worth to notice that every optical module can be operated as an ES or IS
optical switch. As shown in Fig. 1(b) each optical module forwards the input WDM signals from the ToRs in the cluster
by the WSSs to the other ToRs residing in the cluster. Specifically, the 6x4 mm2 fabricated photonic switch chip in Fig.
1(c) integrates 4 optical modules that can been used to implement 4 ES or 4 IS for 4 ToRs intra-cluster or inter-cluster
interconnection. At the input of each module, an 800 μm booster SOA is employed to compensate the 6 dB losses of the
1:4 splitter and partially the AWGs losses at the WSS. The passive 1:4 splitter is realized by cascading 1×2 multimode
interferometer (MMI). Each of the four identical modules processes one of the four WDM inputs and forwards them to
the dedicated outputs. Therefore, two photonics integrated switch chips can be used to implement the 4 ES and 4 IS
switches to interconnect 16 ToRs grouped in 4 clusters.
EXPERIMENTAL SET-UP AND RESULTS
The experimental set-up to assess the fast dynamic control of the optical DCN based on the nanoseconds photonics
integrated switches is shown in Fig. 2. The optical modules of PIC-1 are used to implement the intra-cluster switches IS-
1, IS-2, IS-3, and IS-4 that interconnects the 4 ToRs of cluster 1, cluster 2, cluster 3, and cluster 4, respectively. The
optical modules of PIC-2 are utilized to implement the inter-cluster switches ES-1, ES-2, ES-3, and ES-4 that
interconnects the 4 i-th ToRs of these 4 clusters (1≤i≤4). Note that given the modularity of the architecture (4 ToRs
interconnected by the intra-cluster switch IS, and 4 inter-cluster ToRs connected by the inter-cluster ES), and being all
the optical modules of the PICs the same copy, the assessment of one optical module (IS-1) connecting the 4 ToRs in the
cluster (cluster 1) is representative for all the other intra-cluster and inter-cluster switch modules operation and
performance. Therefore, the dynamic switching operation with optical flow control within the 4 ToRs of cluster 1 has
been investigated for the IS-1.The 4 ToRs are implemented by an FPGA that integrates the packets (labels and payloads)
generation, the electronic buffers, and the optical Flow Control protocol for each distinct ToR. The FPGA is equipped
with 10 Gb/s SFP transceivers at 1525.0nm, 1528.9nm, 1532.9nm and 1536.8nm to generate the distinct optical packets
for the four ToRs, respectively. The optical packets time slot is 600 ns (540 ns payload time and 60 ns guard time). The
four ToRs optical channels are amplified and injected to the unpackaged IS-1 PIC switch module via an EDFA, while the
optical label signals are sent to the FPGA-based switch controllers. After detecting the optical labels and resolving the
packets contention, the switch controller sets the SOA gates to dynamic control the PIC switch.
First, we assess and validate the dynamic switching operation of the IS-1. In particular, we first assess the switching
operation of the WSS1 of the IS-1 module. The switch enabling signals (label control 1-4) generated by the FPGA switch
controller after processing the optical labels of the 4 ToR channels, and the switched channels at the WSS1 (output port
1) are reported in Fig. 3(a). The enabling signals are synchronized with the optical packets and a bias current of 40mA is
applied for the “on” state of the SOA gates. The traces indicate that the optical packets are dynamically switched according
to the FPGA control signals. Bit error rate (BER) curves for the switched 4 ToRs channel inputs are reported in Fig. 3(b).
Fig. 2: Experimental set-up employed for performance evaluation.

OECC/PSC 2019
©IEICE
The back-to-back (B2B) curve is included as reference. Error-free operations with < 1 dB have been measured for ToR 1
(CH 1) and ToR 3 (CH 3) packets, while for ToR 2 (CH2) and TOR 4 (CH 4) packets the penalty is around 2 dB, but still
enough quality to be correctly detected. Secondly, the dynamic switch control operation of the packets to the four WSSs
output ports has also been validated. Packets from ToR 1 has been switched to the 4 output ports by dynamic controlling
the SOA gates of the 4 WSSs. The trace of the label control signals and the switched optical packets at the four outputs
are reported in Fig. 3 (c). The switched packets are detected by the FPGA ToRs and the measured ToR-to-ToR latency is
260.66 ns. Those results confirm that the fast dynamic control operation between ToRs and IS, and validate the switching
operation in space, wavelength, and time domain of the FPGA controlled PIC switch.
The distributed optical Flow Control protocol utilized to prevent the packets loss at the switch is also demonstrated.
Heavy traffic load is generated by the FPGA based ToRs to induce high packets contention. Fig. 3(d) shows the
transmitted optical labels of ToR 1-4, RequestMessage_NextDestination signals (RM), with the packets destined ports
information to the switch controller at every time slot. The FPGA switch controller processes the 4 RM signals and
operates the contention resolution protocol to generate the RequestResponse_NextDestination (RR) signals back to the
ToRs. When there is no packet contention (e.g. RM: 003, 002, 000, 001), the RR (003, 002, 000, 001) signals will be sent
to the corresponding ToRs. While in case of packets contention (e.g. RM: 003, 000, 000, 002, where the packets from the
ToR 1 and ToR 2 have the same destination 000), the RR signals sent back to the ToRs will be 003, 000, 001, 002. As
shown in Fig. 3 (e), when the ToR receives the ACK signal (RR=RM), the ToR will release the optical payload stored in
the buffer and send a new RM signal for the next payloads in the following time slot. When receives a NACK signal (RR
/= RM), the ToR will retransmit the same RM signal and the corresponding optical payload until receiving the ACK
signal meaning the successful forwarding. The monitored statistics (counts of lost and retransmitted packets) at the FPGA
controller shown in Fig. 3(f) confirm that the optical Flow Control protocol prevents any packets loss at the PIC switch.
CONCLUSIONS
The fast dynamic and distributed optical flow control of the optical DCN based on distributed nanoseconds photonics
integrated switches are implemented and experimentally assessed. The proposed photonics switch based DCN can be fast
and dynamically controlled by the optical flow control protocol implemented between the FPGA-based switch controllers
and ToRs. Experimental results indicate that the photonics switch based DCN can switch error-free the 10 Gb/s traffic in
space, wavelength, and time domain with no packets loss and <2dB penalty and 260.66 ns ToR-to-ToR latency. .
ACKNOWLEDGMENT
The authors would like to thank the H2020 Passion (780326) and H2020 Qameleon (780354) projects for partially
supporting this work.
REFERENCES
[1] Cisco Global Cloud Index: Forecast and Methodology, 2016–2021 White Paper, USA. (2016).
[2] W. Xia et al. “A survey on data center networking: Infrastructure and operations” IEEE Commun. surveys & tutorials, 2017.
[3] N. Calabretta et al. “Monolithically integrated WDM cross-connect switch for high-performance optical data center networks”
2017 Optical Fiber Communications Conference and Exhibition (OFC). IEEE, 2017: 1-3.
[4] M. Wang et al. “Low latency and efficient optical flow control for intra data center networks” Optics express(2014), 427-434.
[5] F. Yan et al. “Opsquare: A flat DCN architecture based on flow-controlled optical packet switches” IEEE/OSA Journal of
Optical Communications and Networking, 2017, 9(4): 291-303.
Fig. 3: (a) Traces for WSS1; (b) BER curves for WDM channels input at 10 Gb/s. CH: channel; (c) Traces for 4 outputs; Label requests and
responses at (d) switch controller and (e) ToR; (f) Statistics monitored at switch controller.