Abstract:The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother. In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by developing a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours, collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis. The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.

HEP data analysis using jHepWork and Java

HEP-Frame: an Efficient Tool for Big Data Applications at the LHC

The HTPmod Shiny application enables modeling and visualization of large-scale biological data

Big Data in HEP: A comprehensive use case study

Using Big Data Technologies for HEP Analysis

HepData and JetWeb: HEP data archiving and model validation

Toward real-time data query systems in HEP

HEP Analysis Facility An Approach to Grid Computing

PyHEP.dev 2024 Workshop Summary Report, August 26-30 2024, Aachen, Germany

Interference-filter-stabilized external-cavity diode lasers

HepLean: Digitalising high energy physics

EasyScan_HEP: a tool for connecting programs to scan the parameter space of physics models

Power Profile Monitoring and Tracking Evolution of System-Wide HPC Workloads

Animating collider processes with event-time-frame format

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments

Potential of the Julia programming language for high energy physics computing

A Plugin-Based Software Framework for Data Acquisition and Online Processing

HEP Software Foundation Community White Paper Working Group --- Visualization

A customizable data management framework for high-repetition-rate high-energy-density science

An Operator Library System for HEP Events Processing Based on Hbase Coprocessor.

High Energy Physics Forum for Computational Excellence: Working Group Reports (I. Applications Software II. Software Libraries and Tools III. Systems)