Abstract:Service function chaining (SFC) focuses mainly on deploying various network functions in geographically distributed data centers and providing interconnect routing among them. Traditional (convex optimization-based) SFC algorithms exhibit some disadvantages on the scalability and accuracy. Recently, researches have shown the effectiveness of deep reinforcement learning (DRL) in the field of SFC. However, current DRL-based algorithms possess an extremely large action space, which leads to slow convergence and poor scalability. Some researchers relieve this issue by reformulating the SFC problem, which usually results in low utilization and high cost. To address this issue, we develop a hybrid DRL-based framework which decouples the VNF deployment and flow routing into different modules. In the proposed framework, a DRL agent is only responsible for learning the policy of VNF deployment. We customize the structure of the agent base on deep deterministic policy gradient (DDPG) and adopt several techniques to improve the learning efficiency, such as adaptive parameter noise, wolpertinger policy, and prioritized experience replay. The flow routing is conducted in a game-based module (GBM). We design a decentralized routing algorithm for the GBM to address the scalability. The end-to-end latency of flows is minimized while the resource capacity and location constraints are satisfied. During the learning process of the proposed framework, the DRL agent improves its deployment policy with the reward from the GBM (the value of reward depends on flow routing). Thus, the VNF deployment and flow routing are still jointly optimized. Compared to existing DRL-learning algorithms, the proposed hybrid DRL framework can achieve a lower cost since 1) the action space is significantly reduced due to flow routing decoupling; 2) the flow routing procedure is more efficient (the GBM adopts model-based information, e.g., the gradient). Through trace-driven simulations, we show the efficiency of our algorithm compared to existing DRL-based algorithms.

DRLFluent: A Distributed Co-Simulation Framework Coupling Deep Reinforcement Learning with Ansys-Fluent on High-Performance Computing Systems

DRLinFluids---An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM

Optimal parallelization strategies for active flow control in deep reinforcement learning-based computational fluid dynamics

An efficient intrusive deep reinforcement learning framework for OpenFOAM

Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach

Robust and Adaptive Deep Reinforcement Learning for Enhancing Flow Control around a Square Cylinder with Varying Reynolds Numbers

A dual grid-based deep reinforcement learning and computational fluid dynamics method for flow control and its application to nucleate pool boiling

HybridFlow: A Flexible and Efficient RLHF Framework

DRLinSPH: An open-source platform using deep reinforcement learning and SPHinXsys for fluid-structure-interaction problems

A field programmable gate array-based deep reinforcement learning framework for experimental active flow control and its application in airfoil flow separation elimination

A Hybrid Learning Framework for Service Function Chaining Across Geo-Distributed Data Centers.

Surrogate model-based deep reinforcement learning for experimental study of active flow control of circular cylinder

Deep reinforcement transfer learning for active flow control of a 3D square cylinder under state dimension mismatch

An application of data driven reward of deep reinforcement learning by dynamic mode decomposition in active flow control

Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing

Deep Reinforcement Learning: A New Beacon for Intelligent Active Flow Control

Deep Reinforcement Learning for Active Flow Control around a Circular Cylinder Using Unsteady-mode Plasma Actuators

Accelerating and improving deep reinforcement learning-based active flow control: Transfer training of policy network

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning

Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization