Abstract:Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.

Data transfer performance issues for a web services interface to synchrotron experiments

Binary Data Transfer Performance over High-Latency Networks Using Web Service Attachments

Providing Portlet-Based Client Access to Cima-Enabled Crystallographic Instruments, Sensors, and Data

Data Transfer Performance of Web Service Workflows in the Cloud Environment

Instrument Monitoring, Data Sharing, and Archiving Using Common Instrument Middleware Architecture (CIMA)

Developing CIMA-based cyberinfrastructure for remote access to scientific instruments and collaborative e-research

Developing CIMA based Remote Access for Collaborative e-Research

Integrated System of Heterogeneous Data Based on Web Services

Data Transfer Transmission for Cloud Service Environment of Scientiifc Computing

Demystifying the Performance of Data Transfers in High-Performance Research Networks

Performance Evaluation of Concurrent Multistreamed Web Transport Based on SCTP

Building high-speed roads: Improving performance of SOAP processing for cloud services

A Comparison of Federated Databases with Web Services for the Integration of Bioinformatics Data

BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

Implementing High Performance Remote Method Invocation in CCA

Design of a Common High-Performance Data Acquisition System for Physics Experiments at HIAF

A Framework of Access Performance Optimization under Concurrent Access Environment

Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations

Research on the Architecture and its Implementation for Instrumentation and Measurement Cloud

A Middleware-Based Implementation for Data Integration of Remote Devices

Experiments and Analyses of Data Transfers over Wide-Area Dedicated Connections