Abstract:PyTorch has ascended as a premier machine learning framework, yet it lacks a native and comprehensive library for decision and control tasks suitable for large development teams dealing with complex real-world data and environments. To address this issue, we propose TorchRL, a generalistic control library for PyTorch that provides well-integrated, yet standalone components. We introduce a new and flexible PyTorch primitive, the TensorDict, which facilitates streamlined algorithm development across the many branches of Reinforcement Learning (RL) and control. We provide a detailed description of the building blocks and an extensive overview of the library across domains and tasks. Finally, we experimentally demonstrate its reliability and flexibility and show comparative benchmarks to demonstrate its computational efficiency. TorchRL fosters long-term support and is publicly available on GitHub for greater reproducibility and collaboration within the research community. The code is open-sourced on GitHub.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the lack of a general - purpose library in current decision - making and control tasks that can comprehensively support complex real - world data and environments. Specifically, the author points out that although PyTorch has become one of the top frameworks in the field of machine learning, it lacks a native and comprehensive library for decision - making and control tasks, especially for large - scale development teams. Existing solutions are either too high - level to be reused in different application scenarios, or too low - level to be actually used by non - expert users. This has led to a slow standardization process of decision - making algorithms, especially when dealing with a wide range of application scenarios such as games, robot control, autonomous driving, finance, online advertising bidding, cooling system control, fast matrix multiplication algorithms, and chip design. To address this challenge, the author proposes TorchRL, a general - purpose control library based on PyTorch. TorchRL promotes algorithm development in the fields of reinforcement learning (RL) and control by introducing a new and flexible PyTorch primitive - TensorDict. TensorDict, as a powerful communication class, allows each component of the library to run independently without limiting its scope, while ensuring seamless integration between them, thus flexibly creating algorithms. In addition, TorchRL aims to provide long - term support and is publicly released on GitHub to improve reproducibility and collaboration within the research community. Overall, the goal of TorchRL is to provide an easy - to - use, powerful, efficient, scalable, and widely applicable decision - making tool to meet the needs of this rapidly evolving and highly heterogeneous field.

TorchRL: A data-driven decision-making library for PyTorch

RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Control

rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch

Tianshou: A Highly Modularized Deep Reinforcement Learning Library

Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python

BricksRL: A Platform for Democratizing Robotics and Reinforcement Learning Research and Education with LEGO

TorchDriveEnv: A Reinforcement Learning Benchmark for Autonomous Driving with Reactive, Realistic, and Diverse Non-Playable Characters

torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

lfads-torch: A modular and extensible implementation of latent factor analysis via dynamical systems

XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library

SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning

Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach

TorchDEQ: A Library for Deep Equilibrium Models

Rethinking Decision Transformer via Hierarchical Reinforcement Learning

CytonRL: an Efficient Reinforcement Learning Open-source Toolkit Implemented in C++

PyTorch: An Imperative Style, High-Performance Deep Learning Library

TorchQL: A Programming Framework for Integrity Constraints in Machine Learning

TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions

torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP

TorchMD: A deep learning framework for molecular simulations

TorchOpt: an Efficient Library for Differentiable Optimization.