Data Center Interconnects at 400G and Beyond

Michael Eiselt,Annika Dochhan,Joerg-Peter Elbers

DOI: https://doi.org/10.1109/OECC.2018.8730011

2018-07-31

Abstract:Current trends in Data Center Interconnectivity are considered in the light of increasing traffic and under the constraint of limited cost and power consumption.

Networking and Internet Architecture,Signal Processing

What problem does this paper attempt to address?

Practical Solutions for 400 Gbit/s Data Center Transmission

Annika Dochhan,Nicklas Eiselt,Jinlong Wei,Helmut Griesser,Michael Eiselt,Juan José Vegas Olmos,Idelfonso Tafur Monroy,Jörg-Peter Elbers

DOI: https://doi.org/10.1364/ACPC.2016.AS1B.3

2020-09-22

Abstract:We review three solutions for low-cost data center interconnects with a target reach of up to 80 km. Directly detected DMT, PAM-4 and multi-band CAP are promising modulation schemes, enabling 400 Gbit/s by combining eight channels of 56 Gbit/s.

Signal Processing
Advanced RF/baseband Interconnect Schemes for Inter- and Intra-Ulsi Communications

MCF Chang,I Verbauwhede,C Chien,ZW Xu,JS Kim,JW Ko,Q Gu,BC Lai

DOI: https://doi.org/10.1109/ted.2005.850699

IF: 3.1

2005-01-01

IEEE Transactions on Electron Devices

Abstract:Future inter- and intra-ULSI interconnect systems demand extremely high data rates (up to 100 Gbps/pin or 20-Tbps aggregate) as well as bidirectional multiI/O concurrent service, re-configurable computing/processing architecture, and total compatibility with mainstream silicon system-on-chip and system-in-package technologies. In this paper, we review recent advances in interconnect schemes that promise to meet all of the above system requirements. Unlike traditional wired interconnects based solely on time-division multiple access for data transmission, these new interconnect schemes facilitate the use of additional multiple access techniques including code-division multiple access and frequency-division multiple access to greatly increase bandwidth and channel concurrency as well as to reduce channel latency. The physical transmission line is no longer limited to a direct-coupled metal wire. Rather, it can be accomplished via either wired or wireless mediums through capacitor couplers that reduce the baseband noise and dc power consumption while simplifying the fabrication process by eliminating vertical metal studs needed in three-dimensional ICs. These new advances in interconnect schemes would fundamentally alter the paradigm of ULSI data communications and enable the design of next-generation computing/processing systems.
Industry Trends for 400 Gb/s and Beyond Applications

Yang Yue,Qiang Wang,Jian Yao,Zhen Qu,Jon Anderson

DOI: https://doi.org/10.1109/ICOCN.2019.8934702

2019-01-01

Abstract:After the implementation of 100-Gb/s pluggable optics, the optical communications industry is marching towards 400-Gb/s era. We will review our recent demonstration and study for 400 Gb/s and beyond client and line side optical interconnect.
Experimental Investigation of 400 Gb/s Data Center Interconnect Using Unamplified High-Baud-Rate and High-Order QAM Single-Carrier Signal

Yang Yue,Qiang Wang,Jon Anderson

DOI: https://doi.org/10.3390/app9122455

2019-01-01

Applied Sciences

Abstract:In this article, we review the latest progress on data center interconnect (DCI). We then discuss different perspectives on the 400G pluggable module, including form factor, architecture, digital signal processing (DSP), and module power consumption, following 400G pluggable optics in DCI applications. Next, we experimentally investigate the capacity-reach matrix for high-baud-rate and high-order quadrature amplitude modulation (QAM) single-carrier signals in the unamplified single-mode optical fiber (SMF) link. We show that the 64 GBd 16-QAM, and 64-QAM signals can potentially enable 400 Gb/s and 600 Gb/s DCI application for 40 km and beyond of unamplified fiber link.
Development Trends in Silicon Photonics for Data Centers

Zhiping Zhou,Ruixuan Chen,Xinbai Li,Tiantian Li

DOI: https://doi.org/10.1016/j.yofte.2018.03.009

IF: 2.8

2018-01-01

Optical Fiber Technology

Abstract:An explosive increase in volume of global network and data center traffic requires an interconnection scheme with low cost, high energy efficiency, and high bandwidth capacity. The optical interconnects, especially utilizing silicon photonic platform, have been widely proposed to meet this challenging requirement as they intrinsically feature strong promise to obtain a high bandwidth and power efficient short-reach interconnection in much lower cost. In this review, we will discuss the current development trends in silicon photonics for data-center application with emphasis on reducing cost, lowering energy consumption, and increasing capacity.
Optical Interconnects In High-Density Data Communications

Zp Zhou

DOI: https://doi.org/10.1117/12.402680

2000-01-01

Abstract:Development in both systems and applications are driving semiconductor technology towards gigascale and terascale integration. The electrical interconnection networks are severely limiting this effort due to RC delay, clock skew, cross talk, and electromigration. Many novel communication mechanisms have been proposed, but the optical interconnection remains as the most promising candidate. This paper is to review the optical interconnection in high-density data communications, from the point of view of semiconductor technology, and present the most recent developments and results.
Optical Switching Data Center Networks: Understanding Techniques and Challenges

Xuwei Xue,Shaojuan Zhang,Bingli Guo,Wei Ji,Rui Yin,Bin Chen,Shanguo Huang

DOI: https://doi.org/10.48550/arXiv.2302.05298

2023-01-13

Abstract:Relying on the flexible-access interconnects to the scalable storage and compute resources, data centers deliver critical communications connectivity among numerous servers to support the housed applications and services. To provide the high-speeds and long-distance communications, the data centers have turned to fiber interconnections. With the stringently increased traffic volume, the data centers are then expected to further deploy the optical switches into the systems infrastructure to implement the full optical switching. This paper first summarizes the topologies and traffic characteristics in data centers and analyzes the reasons and importance of moving to optical switching. Recent techniques related to the optical switching, and main challenges limiting the practical deployments of optical switches in data centers are also summarized and reported.

Networking and Internet Architecture,Systems and Control
Numerical and Experimental Study of A Cascaded Microelectromechanical System-Based All-Optical Data Center Interconnect

Qian Kong,Shanguo Huang,Bingli Guo,Xin Li,Min Zhang,Yongli Zhao,Jie Zhang,Wanyi Gu

DOI: https://doi.org/10.1117/1.oe.55.7.076111

IF: 1.3

2016-01-01

Optical Engineering

Abstract:As the scale of the intra-data center network (DCN) grows even larger, the traditional electrical switching has reached a bottle neck in terms of energy consumption, bandwidth provision, and end-to-end latency. Different approaches have been made by employing the optical switch instead of the electrical ones to solve the bandwidth as well as the energy efficiency and the latency problem. We propose a DCN architecture based on cascaded microelectromechanical systems switches for dynamic DCN connectivity provisioning. This architecture provides high port count, which attributes to the demands of the intradata center traffic. Multiple points to one point switching scenario is experimentally demonstrated through this data center interconnect. Numerical simulation is employed to investigate the performance of the proposed architecture. The results show that the blocking probability and latency decrease as the scale of the architecture is upgraded. (C) 2016 Society of Photo-Optical Instrumentation Engineers (SPIE).
Enabling Technologies for Future Data Center Networking: a Primer

Min Chen,Hai Jin,Yonggang Wen,Victor C. M. Leung

DOI: https://doi.org/10.1109/mnet.2013.6574659

IF: 10.294

2013-01-01

IEEE Network

Abstract:The increasing adoption of cloud services is demanding the deployment of more data centers. Data centers typically house a huge amount of storage and computing resources, in turn dictating better networking technologies to connect the large number of computing and storage nodes. Data center networking (DCN) is an emerging field to study networking challenges in data centers. In this article, we present a survey on enabling DCN technologies for future cloud infrastructures through which the huge amount of resources in data centers can be efficiently managed. Specifically, we start with a detailed investigation of the architecture, technologies, and design principles for future DCN. Following that, we highlight some of the design challenges and open issues that should be addressed for future DCN to improve its energy efficiency and increase its throughput while lowering its cost.
Beyond 100 Gb/S: Capacity, Flexibility, and Network Optimization

Kim Roberts,Qunbi Zhuge,Inder Monga,Sebastien Gareau,Charles Laperle

DOI: https://doi.org/10.1364/jocn.9.000c12

IF: 4.508

2017-01-01

Journal of Optical Communications and Networking

Abstract:In this paper, we discuss building blocks that enable the exploitation of optical capacities beyond 100 Gb/s. Optical networks will benefit from more flexibility and agility in their network elements, especially from coherent transceivers. To achieve capacities of 400 Gb/s and more, coherent transceivers will operate at higher symbol rates. This will be made possible with higher bandwidth components using new electro-optic technologies implemented with indium phosphide and silicon photonics. Digital signal processing will benefit from new algorithms. Multi-dimensional modulation, of which some formats are already in existence in current flexible coherent transceivers, will provide improved tolerance to noise and fiber nonlinearities. Constellation shaping will further improve these tolerances while allowing a finer granularity in the selection of capacity. Frequency-division multiplexing will also provide improved tolerance to the nonlinear characteristics of fibers. Algorithms with reduced computation complexity will allow the implementation, at speeds, of direct pre-compensation of nonlinear propagation effects. Advancement in forward error correction will shrink the performance gap with Shannon's limit. At the network control and management level, new tools are being developed to achieve a more efficient utilization of networks. This will also allow for network virtualization, orchestration, and management. Finally, FlexEthernet and FlexOTN will be put in place to allow network operators to optimize capacity in their optical transport networks without manual changes to the client hardware.
Passive Optical Interconnects at Top of the Rack: Offering High Energy Efficiency for Datacenters

Yu Gong,Xuezhi Hong,Yang Lu,Sailing He,Jiajia Chen

DOI: https://doi.org/10.1364/oe.23.007957

IF: 3.8

2015-01-01

Optics Express

Abstract:This paper introduces a new concept, namely passive optical interconnect at top of the rack in the datacenter networks, and investigates several architectures, which use only passive optical components to interconnect different servers. In such a manner, the proposed schemes are able to offer higher bandwidth and significantly improve energy efficiency compared to their electronic counterpart that is based on commodity switches. The proposed passive optical interconnect schemes are experimentally demonstrated in order to validate the transmission performance. Besides, an assessment in terms of energy consumption and cost has also been carried out, which shows our proposed concept can significantly outperform the conventional commodity switches on energy efficiency while keeping the cost in the similar level.
On the Performance of a Scalable Optical Switching Architecture for Flat Intercluster Data Center Network with Centralized Control

Qian Kong,Shanguo Huang,Yu Zhou,Bingli Guo,Yongli Zhao,Min Zhang,Jie Zhang,Wanyi Gu

DOI: https://doi.org/10.1117/1.oe.53.7.075101

IF: 1.3

2014-01-01

Optical Engineering

Abstract:Data centers have to sustain the rapid growth of data traffic due to the increasing demand of band-width-hungry Internet services. The current fat tree topology causes communication bottlenecks in the server interaction process, resulting in power-hungry O-E-O conversions that limit the minimum latency and the power efficiency of these systems. As a result, recent efforts have advocated that all optical data center networks (DCNs) have the capability to adapt to traffic requirements on demand. We present the design, implementation, and evaluation of a cascaded microelectromechanical systems switches-based DCN architecture which dynamically changes its topology and link capacities, thereby achieving unprecedented flexibility to adapt to dynamic traffic patterns. We analyze it under a data center traffic model to determine its suitability for this type of environment. The proposed architecture can be scaled to 3300 input/output ports by available experimental components with low blocking probability and latency. The blocking probability and latency are about 0.03 and 72 ms at a moderate traffic load for 32 input/output ports based on our numerical results, which are much smaller than the results for 4 input/output ports which are 0.13 and 235 ms, respectively. (c) 2014 Society of Photo-Optical Instrumentation Engineers (SPIE)
Flexible bandwidth 448 Gb/s DMT Transmission for Next Generation Data Center Inter-Connects

Annika Dochhan,Helmut Grießer,Michael Eiselt,Jörg-Peter Elbers

DOI: https://doi.org/10.1109/ECOC.2014.6964048

2020-09-21

Abstract:We experimentally evaluate a flexible DMT system using 4 to 8 50-GHz-grid C-band channels to transmit 448 Gb/s over up to 240 km SSMF. VSB filtering enabled by detuned lasers significantly reduces the impact of chromatic dispersion.

Signal Processing
Solutions for 80 km DWDM Systems

Annika Dochhan,Helmut Griesser,Michael Eiselt,Jörg-Peter Elbers

DOI: https://doi.org/10.48550/arXiv.2009.10570

2020-09-22

Abstract:We review currently discussed solutions for 80 km DWDM transmission targeting inter data-center connections at 100G and 400G line rates. PDM-64QAM, PAM4 and DMT are investigated, while the focus lies on directly detected solutions.

Signal Processing
Guest Editorial: Special Issue on Optical Interconnects

Tian Gu,Lionel Kimerling,Ashkan Seyedi,Thomas Schrans,Jurgen Michel,Daoxin Dai,Peter O'Brien

DOI: https://doi.org/10.1109/JLT.2020.3004669

IF: 4.7

2020-01-01

Journal of Lightwave Technology

Abstract:The articles in this special section focus on optical interconnects. These papers present the most recent developments in this field with extensiveness and depth, while ensuring the inclusion of the highest quality work. Large-scale data centers and high-performance computers (HPC) are becoming increasingly limited by the capacity of interconnects due to the rapidly increasing power consumption and bandwidth demands. As a result of the continuing bandwidth scaling, optical interconnects (OI) have been implemented at ever decreasing communication distances to replace electrical interconnects. Over the past decade, core switches and routers in data centers and HPC systems have adopted fiber optics to deliver the I/O performance between racks in multi-chassis configurations. Meanwhile, the further penetration of OI into shorter distances, such as the board and chip levels, poses new challenges on photonic integrated circuits (PICs) for continued performance and cost scaling and further requires innovations in integration and packaging.
Scalable and Cost-Effective Interconnection of Data-Center Servers Using Dual Server Ports

Dan Li,Chuanxiong Guo,Haitao Wu,Kun Tan,Yongguang Zhang,Songwu Lu,Jianping Wu

DOI: https://doi.org/10.1109/tnet.2010.2053718

2011-01-01

IEEE/ACM Transactions on Networking

Abstract:The goal of data-center networking is to interconnect a large number of server machines with low equipment cost while providing high network capacity and high bisection width. It is well understood that the current practice where servers are connected by a tree hierarchy of network switches cannot meet these requirements. In this paper, we explore a new server-interconnection structure. We observe that the commodity server machines used in today's data centers usually come with two built-in Ethernet ports, one for network connection and the other left for backup purposes. We believe that if both ports are actively used in network connections, we can build a scalable, cost-effective interconnection structure without either the expensive higher-level large switches or any additional hardware on servers. We design such a networking structure called FiConn. Although the server node degree is only 2 in this structure, we have proven that FiConn is highly scalable to encompass hundreds of thousands of servers with low diameter and high bisection width. We have developed a low-overhead traffic-aware routing mechanism to improve effective link utilization based on dynamic traffic state. We have also proposed how to incrementally deploy FiConn.
Energy Consumption of Hybrid Data Center Networks.

Joel Reginald Dodoo,Weiqiang Sun,Feng Zhu,Weisheng Hu

DOI: https://doi.org/10.23919/cnsm46954.2019.9012694

2019-01-01

Abstract:Existing data center networks (DCNs) based on only electronic packet or all-optical switching still pose an exponential increase in power consumption and cost due to the current high demand for digital data and sustainability issues. The recent development of hybrid DCN prototypes is a promising solution offering relatively higher data throughput, low latency, reduction in cost and energy consumption. This paper explores the frontier of hybrid DCN with a special focus on energy consumption and cost. We evaluate the energy consumption per bit and the greenhouse gas (GHG) emission per year of three hybrid switching systems as compared with an optical point to point (ptp) network, results show that the hybrid switching systems will consume less energy per bit and are likely to emit less GHG annually. We present feasibility analysis on the energy consumption and cost of some hybrid DCN prototypes. Evaluation results show that Helios-like and Hybrid Optical Switching-like prototypes achieve a power usage effectiveness (PUE) value lower than 1.2, an index which represents a very efficient level of energy performance in a data center network.
Optical Comb Enabled Flexible-Bandwidth Single-Carrier Frequency-Division-multiplexed-based Intra-Data Centre Interconnect

Qian Kong,Yafeng Zhan,Chaowei Duan

DOI: https://doi.org/10.1049/iet-com.2018.5061

IF: 1.345

2018-01-01

IET Communications

Abstract:In this study, the authors propose a novel intra-data centre network (DCN) architecture based on cascaded micro-electro-mechanical system switches for dynamic DCN connectivity provisioning. Through combining the single-carrier frequency-division-multiplexed with the proposed architecture, this structure can achieve unprecedented flexibility in dealing with both mice and elephant flow in the DCN. Multiple-input multiple-output switching is experimentally demonstrated through intra- and inter-data centre switching scenarios. A numerical investigation is also performed to evaluate the performance of the fixed-grid transceivers, flex-grid transceivers and mixed-fixed/flex grid transceivers in the same switching scenario. The results show that the flex-grid transceivers significantly reduced the service latency and blocking probability compared to the fixed-grid transceivers. The proposed architecture would be better applied in the field of all-optical DCNs.
Sub-THz/THz Interconnect, Complement to Electrical and Optical Interconnects: Addressing Fundamental Challenges Related to Communication Distances

Qun Jane Gu

DOI: https://doi.org/10.1109/mssc.2020.3021836

2020-01-01

Abstract:The big data era has witnessed expansion of information traffic at a 26% compound annual growth rate (CAGR) during one decade, as shown in Figure 1(a) [1], [2]. Rising data rates magnify hierarchically down to shorter communication distances. That is, data rate increases are exchanged from interdata centers to interrack, interboard, interchip, and intrachip scenarios. For almost two decades, microprocessor speeds have been stalled due to unsustainable power consumption [4], resulting in multicore computer architectures. Although the multicore architecture mitigates the data processing power consumption requirement, it demands higher data communication rates from core to core, due to increasingly complex functionalities and systems. Figure 1(b) presents the input?output (I/O) bandwidth and I/O pin number evolution through time [5], [6]. The I/O bandwidth increases by approximately 2 every two years or 10 every five years, while the I/O pin number grows at a much smaller rate, 1.7 every five years, because of process and mechanical constraints. This results in a widening gap between I/O bandwidth requirements and capabilities, notoriously named the interconnect gap, which has been a major and long-standing challenge for more than a decade.

English Else
Next-Generation Optically-Interconnected High-Performance Data Centers

Wenjia Zhang,Howard Wang,Keren Bergman

DOI: https://doi.org/10.1109/JLT.2012.2212696

2012-01-01

Abstract:The communication requirements imposed by the unabated growth in both the size and computational density of modern data centers will soon outpace the fundamental capabilities of conventional electronic interconnection networks. Optical interconnects represent a potentially disruptive technology that can simultaneously satisfy the throughput, latency, and energy demands of these next-generation systems. In this paper, we propose an end-to-end photonic networking platform for future optically-interconnected data center networks. A reconfigurable hybrid photonic network building block and a protocol-agnostic optical network interface comprise the key elements of our proposed system. A modular optical switch fabric prototyping platform is designed and implemented to support a wide variety of potential photonic technologies. We successfully demonstrate nanosecond-scale optical packet-switching at 1-Tb/s per port (40-Gb/s ×25λs) through our prototype, configured as a 4×4 switch, and confirm error-free transmission across all wavelengths. In order to flexibly allocate appropriate bandwidth for heterogeneous applications, we further implement and evaluate a novel wavelength-reconfigurable optical packet- and circuit-switch by utilizing our optical switch platform. Finally, we detail our implementation of an optical network interface card (O-NIC) designed to bridge the gap between the well-established protocol layers utilized in current networks and the physical layer of the proposed photonic networks and demonstrate its functionality by the streaming of high-definition video through the end-to-end optical network.

Data Center Interconnects at 400G and Beyond

Practical Solutions for 400 Gbit/s Data Center Transmission

Advanced RF/baseband Interconnect Schemes for Inter- and Intra-Ulsi Communications

Industry Trends for 400 Gb/s and Beyond Applications

Experimental Investigation of 400 Gb/s Data Center Interconnect Using Unamplified High-Baud-Rate and High-Order QAM Single-Carrier Signal

Development Trends in Silicon Photonics for Data Centers

Optical Interconnects In High-Density Data Communications

Optical Switching Data Center Networks: Understanding Techniques and Challenges

Numerical and Experimental Study of A Cascaded Microelectromechanical System-Based All-Optical Data Center Interconnect

Enabling Technologies for Future Data Center Networking: a Primer

Beyond 100 Gb/S: Capacity, Flexibility, and Network Optimization

Passive Optical Interconnects at Top of the Rack: Offering High Energy Efficiency for Datacenters

On the Performance of a Scalable Optical Switching Architecture for Flat Intercluster Data Center Network with Centralized Control

Flexible bandwidth 448 Gb/s DMT Transmission for Next Generation Data Center Inter-Connects

Solutions for 80 km DWDM Systems

Guest Editorial: Special Issue on Optical Interconnects

Scalable and Cost-Effective Interconnection of Data-Center Servers Using Dual Server Ports

Energy Consumption of Hybrid Data Center Networks.

Optical Comb Enabled Flexible-Bandwidth Single-Carrier Frequency-Division-multiplexed-based Intra-Data Centre Interconnect

Sub-THz/THz Interconnect, Complement to Electrical and Optical Interconnects: Addressing Fundamental Challenges Related to Communication Distances

Next-Generation Optically-Interconnected High-Performance Data Centers