Abstract:Bioinformatics pipelines depend on shared POSIX filesystems for its input, output and intermediate data storage. Containerization makes it more difficult for the workloads to access the shared file systems. In our previous study, we were able to run both ML and non-ML pipelines on Kubeflow successfully. However, the storage solutions were complex and less optimal. This is because there are no established resource types to represent the concept of data source on Kubernetes. More and more applications are running on Kubernetes for batch processing. End users are burdened with configuring and optimising the data access, which is what we have experienced before. In this article, we are introducing a new concept of Dataset and its corresponding resource as a native Kubernetes object. We have leveraged the Dataset Lifecycle Framework which takes care of all the low-level details about data access in Kubernetes pods. Its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Together, they manage the entire lifecycle of the custom resource Dataset. We use Dataset Lifecycle Framework to serve data from object stores to both ML and non-ML pipelines running on Kubeflow. With DLF, we make training data fed into ML models directly without being downloaded to the local disks, which makes the input scalable. We have enhanced the durability of training metadata by storing it into a dataset, which also simplifies the set up of the Tensorboard, separated from the notebook server. For the non-ML pipeline, we have simplified the 1000 Genome Project pipeline with datasets injected into the pipeline dynamically. In addition, our preliminary results indicate that the pluggable caching mechanism can improve the performance significantly.

Feature Stores: A Key Enabler for Feature Reusability and Availability Across Machine Learning Pipelines

Managed Geo-Distributed Feature Store: Architecture and System Design

Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems

An Exploratory Study on Machine Learning Model Stores

On Designing Data Models for Energy Feature Stores

Modeling Quality and Machine Learning Pipelines through Extended Feature Models

Smart Data Placement Using Storage-as-a-Service Model for Big Data Pipelines

Dataset Lifecycle Framework and its applications in Bioinformatics

Py_FS: A Python Package for Feature Selection Using Meta-Heuristic Optimization Algorithms

Towards Observability for Production Machine Learning Pipelines

FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning

High Performance I/O For Large Scale Deep Learning

Statistically Enhanced Learning: a feature engineering framework to boost (any) learning algorithms

MvFS: Multi-view Feature Selection for Recommender System

A comprehensive survey on feature selection in the various fields of machine learning

FeatureEnVi: Visual Analytics for Feature Engineering Using Stepwise Selection and Semi-Automatic Extraction Approaches

MetaFS: An Effective Wrapper Feature Selection via Meta Learning

Single Sample Feature Importance: An Interpretable Algorithm for Low-Level Feature Analysis

Dynamic and Adaptive Feature Generation with LLM

AFS: An Attention-based mechanism for Supervised Feature Selection

D-FS: A Novel Integration Method of Discretization and Feature Selection