Abstract:Nowadays more and more data have been generated every day in some enterprises such as facebook and google. These data need to be collected and analyzed in time. So the speed of transmitting data must be very high and the latency must be very low. Hadoop is applied in these enterprises and they use several data centers to store and process these data. But if the amount of data is growing fast or we will use only one data center then the bandwidth of the Ethernet Hadoop Distributed File System (HDFS) using cannot meet the need. The bandwidth of the Ethernet is going to become the performance bottleneck of HDFS. In order to solve this problem we will introduce a relatively new switched fabric communication link - Infiniband in this paper. Based on Infiniband we have designed a new communication mechanism of HDFS and implemented it by modifying the code of HDFS. We use remote direct memory access (RDMA) to send and receive data rather than socket. The new HDFS will not use original stream mode to transmit data. Instead it will dynamically expand buffer and use changeable threshold. In this way the new HDFS will make CPU idle and improve performance. Unlike IPoIB which only uses Infiniband hardware device, our optimized HDFS is not only based on Infiniband hardware but also changes the code of HDFS to use RDMA. Our HDFS uses socket to transmit control message and RDMA to transmit data to make full use of the bandwidth of Infiniband. So applying the Infiniband with RDMA network bandwidth has not been the performance bottleneck of HDFS any more. According to the experiment results we have found that the network bandwidth of HDFS over Infiniband is 60 percent higher than the Ethernet and our optimized HDFS has much better performance than the HDFS over the Ethernet. On the other hand, the performance of our HDFS is also higher than the one which only use IPoIB. © 2013 Springer-Verlag.

Bandwidth Performance Analysis And Evaluation Of Quad-Data Rate Infiniband

Advanced RF/baseband Interconnect Schemes for Inter- and Intra-Ulsi Communications

Design and Implementation of HDFS over Infiniband with RDMA.

Receiver-Driven Congestion Control for InfiniBand.

IBShare : A Dynamic IB Bandwidth Allocation Based on Congestion Control ⋆

Research of Infiniband Architecture

Experimental evaluation of high speed impulse radio UWB interference on WiMAX narrowband systems

The Analyse of SDP in InfiniBand Architecture

Improving the Performance of MongoDB with RDMA.

Performance Characterization of SmartNIC NVMe-over-Fabrics Target Offloading

Securing InfiniBand Networks with the Bluefield-2 Data Processing Unit

HyQ: Hybrid I/O Queue Architecture for NVMe over Fabrics to Enable High- Performance Hardware Offloading

RDMA Performance Isolation With Justitia

Modeling and Analyzing the Performance of High-Speed Packet I/O

FlowStar: Fast Convergence Per-Flow State Accurate Congestion Control for InfiniBand

Passive Equalizer Design on High Density Interconnects for 25gbps High Speed Io and Beyond

Research and Implementation of Iscsi HBA Based on Intelligent I/O

RB2: Narrow the Gap Between RDMA Abstraction and Performance Via a Middle Layer

Data Transfer Performance Analysis and Enhancement of Critical 3D Interconnects in a 3D SiP Based on Communication Channel Modeling Methodology

Qswitch: Dynamical Off-Chip Bandwidth Allocation Between Local and Remote Accesses.

Adaptive and Scalable Optimizations for High Performance SR-IOV