Abstract:There hardly exists a general solver that is efficient for scheduling problems due to their diversity and complexity. In this study, we develop a two-stage framework, in which reinforcement learning (RL) and traditional operations research (OR) algorithms are combined together to efficiently deal with complex scheduling problems. The scheduling problem is solved in two stages, including a finite Markov decision process (MDP) and a mixed-integer programming process, respectively. This offers a novel and general paradigm that combines RL with OR approaches to solving scheduling problems, which leverages the respective strengths of RL and OR: The MDP narrows down the search space of the original problem through an RL method, while the mixed-integer programming process is settled by an OR algorithm. These two stages are performed iteratively and interactively until the termination criterion has been met. Under this idea, two implementation versions of the combination methods of RL and OR are put forward. The agile Earth observation satellite scheduling problem is selected as an example to demonstrate the effectiveness of the proposed scheduling framework and methods. The convergence and generalization capability of the methods are verified by the performance of training scenarios, while the efficiency and accuracy are tested in 50 untrained scenarios. The results show that the proposed algorithms could stably and efficiently obtain satisfactory scheduling schemes for agile Earth observation satellite scheduling problems. In addition, it can be found that RL-based optimization algorithms have stronger scalability than non-learning algorithms. This work reveals the advantage of combining reinforcement learning methods with heuristic methods or mathematical programming methods for solving complex combinatorial optimization problems.

Framework for Modeling and Optimization of On-Orbit Servicing Operations under Demand Uncertainties

On-Orbit Servicing Optimization Framework with High- and Low-Thrust Propulsion Tradeoff

Optimal Orbit Design and Mission Scheduling for Sun-Synchronous Orbit On-orbit Refueling System

An optimization framework with improved auction-based initialization for highly constrained on-orbit servicing mission planning

Optimal Deployment of Heterogeneous Microsatellite Constellation Based on Kuhn-Munkres and Simulated Annealing Algorithms

Hierarchical Framework for Space Exploration Campaign Schedule Optimization

Integrated Space Logistics Mission Planning and Spacecraft Design with Mixed-Integer Nonlinear Programming

Modeling and Optimization for Space Logistics: Review of State of the Art

Enriching On-orbit Servicing Mission Planning with Debris Recycling and In-space Additive Manufacturing

GEO Satellite On-Orbit Refueling and Debris Removal Hybrid Mission Planning under Uncertainty

Multi-Fidelity Space Mission Planning and Infrastructure Design Framework for Space Resource Logistics

Distributionally Robust Optimization of On-Orbit Resource Scheduling for Remote Sensing in Space-Air-Ground Integrated 6G Networks

Event-Driven Network Model for Space Mission Optimization with High-Thrust and Low-Thrust Spacecraft

On-Orbit Servicing System Architectures for Proliferated Low-Earth-Orbit Constellations

Optimal Launch Manifesting for Space Freight Forwarders

Optimizing Participant Selection for Fault-Tolerant Decision Making in Orbit Using Mixed Integer Linear Programming

Optimal Satellite Constellation Spare Strategy Using Multi-Echelon Inventory Control

Flexibility Management for Space Logistics via Decision Rules

A Two-stage Framework and Reinforcement Learning-based Optimization Algorithms for Complex Scheduling Problems

Robust Optimization, Structure/Control co-design, Distributed Optimization, Monolithic Optimization, Robust Control, Parametric Uncertainty