Reinforcement learning for graph theory, I. Reimplementation of Wagner's approach

Mohammad Ghebleh,Salem Al-Yakoob,Ali Kanso,Dragan Stevanovic

2024-03-30

Abstract:We reimplement here the recent approach of Adam Zsolt Wagner [

Combinatorics

What problem does this paper attempt to address?

Reinforcement learning for graph theory, II. Small Ramsey numbers

Mohammad Ghebleh,Salem Al-Yakoob,Ali Kanso,Dragan Stevanović

2024-03-29

Abstract:We describe here how the recent Wagner's approach for applying reinforcement learning to construct examples in graph theory can be used in the search for critical graphs for small Ramsey numbers. We illustrate this application by providing lower bounds for the small Ramsey numbers $R(K_{2,5}, K_{3,5})$, $R(B_3, B_6)$ and $R(B_4, B_5)$ and by improving the lower known bound for $R(W_5, W_7)$.

Mathematics
A Systematization of the Wagner Framework: Graph Theory Conjectures and Reinforcement Learning

Flora Angileri,Giulia Lombardi,Andrea Fois,Renato Faraone,Carlo Metta,Michele Salvi,Luigi Amedeo Bianchi,Marco Fantozzi,Silvia Giulia Galfrè,Daniele Pavesi,Maurizio Parton,Francesco Morandin

2024-09-17

Abstract:In 2021, Adam Zsolt Wagner proposed an approach to disprove conjectures in graph theory using Reinforcement Learning (RL). Wagner's idea can be framed as follows: consider a conjecture, such as a certain quantity f(G) < 0 for every graph G; one can then play a single-player graph-building game, where at each turn the player decides whether to add an edge or not. The game ends when all edges have been considered, resulting in a certain graph G_T, and f(G_T) is the final score of the game; RL is then used to maximize this score. This brilliant idea is as simple as innovative, and it lends itself to systematic generalization. Several different single-player graph-building games can be employed, along with various RL algorithms. Moreover, RL maximizes the cumulative reward, allowing for step-by-step rewards instead of a single final score, provided the final cumulative reward represents the quantity of interest f(G_T). In this paper, we discuss these and various other choices that can be significant in Wagner's framework. As a contribution to this systematization, we present four distinct single-player graph-building games. Each game employs both a step-by-step reward system and a single final score. We also propose a principled approach to select the most suitable neural network architecture for any given conjecture, and introduce a new dataset of graphs labeled with their Laplacian spectra. Furthermore, we provide a counterexample for a conjecture regarding the sum of the matching number and the spectral radius, which is simpler than the example provided in Wagner's original paper. The games have been implemented as environments in the Gymnasium framework, and along with the dataset, are available as open-source supplementary materials.

Machine Learning
Graph Representations for Reinforcement Learning

Esteban Schab,Carla Casanova,Fabiana Piccoli

DOI: https://doi.org/10.24215/16666038.24.e03

IF: 1.871

2024-04-22

Journal of Computer Science and Technology

Abstract:Graph analysis is becoming increasingly important due to the expressive power of graph models and the efficient algorithms available for processing them. Reinforcement Learning is one domain that could benefit from advancements in graph analysis, given that a learning agent may be integrated into an environment that can be represented as a graph. Nevertheless, the structural irregularity of graphs and the lack of prior labels make it difficult to integrate such a model into modern Reinforcement Learning frameworks that rely on artificial neural networks. Graph embedding enables the learning of low-dimensional vector representations that are more suited for machine learning algorithms, while retaining essential graph features. This paper presents a framework for evaluating graph embedding algorithms and their ability to preserve the structure and relevant features of graphs by means of an internal validation metric, without resorting to subsequent tasks that require labels for training. Based on this framework, three defined algorithms that meet the necessary requirements for solving a specific problem of Reinforcement Learning in graphs are selected, analyzed, and compared. These algorithms are Graph2Vec, GL2Vec, and Wavelet Characteristics, with the latter two demonstrating superior performance.

computer science, software engineering, hardware & architecture
Graph learning-based generation of abstractions for reinforcement learning

Xue, Yuan,Kudenko, Daniel,Khosla, Megha

DOI: https://doi.org/10.1007/s00521-023-08211-x

2023-02-10

Neural Computing and Applications

Abstract:The application of reinforcement learning (RL) algorithms is often hindered by the combinatorial explosion of the state space. Previous works have leveraged abstractions which condense large state spaces to find tractable solutions. However, they assumed that the abstractions are provided by a domain expert. In this work, we propose a new approach to automatically construct abstract Markov decision processes (AMDPs) for potential-based reward shaping to improve the sample efficiency of RL algorithms. Our approach to constructing abstract states is inspired by graph representation learning methods , it effectively encodes the topological and reward structure of the ground-level MDP. We perform large-scale quantitative experiments on a range of navigation and gathering tasks under both stationary and stochastic settings. Our approach shows improvements of up to 8.5 times in sample efficiency and up to 3 times in run time over the baseline approach. Besides, with our qualitative analyses of the generated AMDPs, we are able to visually demonstrate the capability of our approach to preserve the topological and reward structure of the ground-level MDP.

computer science, artificial intelligence
Revisiting Random Walks for Learning on Graphs

Jinwoo Kim,Olga Zaghen,Ayhan Suleymanzade,Youngmin Ryou,Seunghoon Hong

2024-07-01

Abstract:We revisit a simple idea for machine learning on graphs, where a random walk on a graph produces a machine-readable record, and this record is processed by a deep neural network to directly make vertex-level or graph-level predictions. We refer to these stochastic machines as random walk neural networks, and show that we can design them to be isomorphism invariant while capable of universal approximation of graph functions in probability. A useful finding is that almost any kind of record of random walk guarantees probabilistic invariance as long as the vertices are anonymized. This enables us to record random walks in plain text and adopt a language model to read these text records to solve graph tasks. We further establish a parallelism to message passing neural networks using tools from Markov chain theory, and show that over-smoothing in message passing is alleviated by construction in random walk neural networks, while over-squashing manifests as probabilistic under-reaching. We show that random walk neural networks based on pre-trained language models can solve several hard problems on graphs, such as separating strongly regular graphs where the 3-WL test fails, counting substructures, and transductive classification on arXiv citation network without training. Code is available at <a class="link-external link-https" href="https://github.com/jw9730/random-walk" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence
Reinforcement Learning Discovers Efficient Decentralized Graph Path Search Strategies

Alexei Pisacane,Victor-Alexandru Darvariu,Mirco Musolesi

2024-09-12

Abstract:Graph path search is a classic computer science problem that has been recently approached with Reinforcement Learning (RL) due to its potential to outperform prior methods. Existing RL techniques typically assume a global view of the network, which is not suitable for large-scale, dynamic, and privacy-sensitive settings. An area of particular interest is search in social networks due to its numerous applications. Inspired by seminal work in experimental sociology, which showed that decentralized yet efficient search is possible in social networks, we frame the problem as a collaborative task between multiple agents equipped with a limited local view of the network. We propose a multi-agent approach for graph path search that successfully leverages both homophily and structural heterogeneity. Our experiments, carried out over synthetic and real-world social networks, demonstrate that our model significantly outperforms learned and heuristic baselines. Furthermore, our results show that meaningful embeddings for graph navigation can be constructed using reward-driven learning.

Machine Learning,Artificial Intelligence,Multiagent Systems,Social and Information Networks
Towards Generalizability of Multi-Agent Reinforcement Learning in Graphs with Recurrent Message Passing

Jannis Weil,Zhenghua Bao,Osama Abboud,Tobias Meuser

2024-06-04

Abstract:Graph-based environments pose unique challenges to multi-agent reinforcement learning. In decentralized approaches, agents operate within a given graph and make decisions based on partial or outdated observations. The size of the observed neighborhood limits the generalizability to different graphs and affects the reactivity of agents, the quality of the selected actions, and the communication overhead. This work focuses on generalizability and resolves the trade-off in observed neighborhood size with a continuous information flow in the whole graph. We propose a recurrent message-passing model that iterates with the environment's steps and allows nodes to create a global representation of the graph by exchanging messages with their neighbors. Agents receive the resulting learned graph observations based on their location in the graph. Our approach can be used in a decentralized manner at runtime and in combination with a reinforcement learning algorithm of choice. We evaluate our method across 1000 diverse graphs in the context of routing in communication networks and find that it enables agents to generalize and adapt to changes in the graph.

Multiagent Systems,Artificial Intelligence
Deep Reinforcement Learning of Graph Matching.

Chang Liu,Runzhong Wang,Zetian Jiang,Junchi Yan

2020-01-01

Abstract:Graph matching (GM) under node and pairwise constraints has been a building block in areas from combinatorial optimization, data mining to computer vision, for effective structural representation and association. We present a reinforcement learning solver for GM i.e. RGM that seeks the node correspondence between pairwise graphs, whereby the node embedding model on the association graph is learned to sequentially find the node-to-node matching. Our method differs from the previous deep graph matching model in the sense that they are focused on the front-end feature extraction and affinity function learning, while our method aims to learn the back-end decision making given the affinity objective function whether obtained by learning or not. Such an objective function maximization setting naturally fits with the reinforcement learning mechanism, of which the learning procedure is label-free. These features make it more suitable for practical usage. Extensive experimental results on both synthetic datasets, Willow Object dataset, Pascal VOC dataset, and QAPLIB showcase superior performance regarding both matching accuracy and efficiency. To our best knowledge, this is the first deep reinforcement learning solver for graph matching.
Graph Reinforcement Learning for Network Control via Bi-Level Optimization

Daniele Gammelli,James Harrison,Kaidi Yang,Marco Pavone,Filipe Rodrigues,Francisco C. Pereira

2023-05-16

Abstract:Optimization problems over dynamic networks have been extensively studied and widely used in the past decades to formulate numerous real-world problems. However, (1) traditional optimization-based approaches do not scale to large networks, and (2) the design of good heuristics or approximation algorithms often requires significant manual trial-and-error. In this work, we argue that data-driven strategies can automate this process and learn efficient algorithms without compromising optimality. To do so, we present network control problems through the lens of reinforcement learning and propose a graph network-based framework to handle a broad class of problems. Instead of naively computing actions over high-dimensional graph elements, e.g., edges, we propose a bi-level formulation where we (1) specify a desired next state via RL, and (2) solve a convex program to best achieve it, leading to drastically improved scalability and performance. We further highlight a collection of desirable features to system designers, investigate design decisions, and present experiments on real-world control problems showing the utility, scalability, and flexibility of our framework.

Machine Learning,Systems and Control,Optimization and Control
Reinforced Continual Learning for Graphs

Appan Rakaraddi,Siew Kei Lam,Mahardhika Pratama,Marcus De Carvalho

DOI: https://doi.org/10.48550/arXiv.2209.01556

2022-09-04

Abstract:Graph Neural Networks (GNNs) have become the backbone for a myriad of tasks pertaining to graphs and similar topological data structures. While many works have been established in domains related to node and graph classification/regression tasks, they mostly deal with a single task. Continual learning on graphs is largely unexplored and existing graph continual learning approaches are limited to the task-incremental learning scenarios. This paper proposes a graph continual learning strategy that combines the architecture-based and memory-based approaches. The structural learning strategy is driven by reinforcement learning, where a controller network is trained in such a way to determine an optimal number of nodes to be added/pruned from the base network when new tasks are observed, thus assuring sufficient network capacities. The parameter learning strategy is underpinned by the concept of Dark Experience replay method to cope with the catastrophic forgetting problem. Our approach is numerically validated with several graph continual learning benchmark problems in both task-incremental learning and class-incremental learning settings. Compared to recently published works, our approach demonstrates improved performance in both the settings. The implementation code can be found at \url{<a class="link-external link-https" href="https://github.com/codexhammer/gcl" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Artificial Intelligence
Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective

Victor-Alexandru Darvariu,Stephen Hailes,Mirco Musolesi

2024-08-20

Abstract:Graphs are a natural representation for systems based on relations between connected entities. Combinatorial optimization problems, which arise when considering an objective function related to a process of interest on discrete structures, are often challenging due to the rapid growth of the solution space. The trial-and-error paradigm of Reinforcement Learning has recently emerged as a promising alternative to traditional methods, such as exact algorithms and (meta)heuristics, for discovering better decision-making strategies in a variety of disciplines including chemistry, computer science, and statistics. Despite the fact that they arose in markedly different fields, these techniques share significant commonalities. Therefore, we set out to synthesize this work in a unifying perspective that we term Graph Reinforcement Learning, interpreting it as a constructive decision-making method for graph problems. After covering the relevant technical background, we review works along the dividing line of whether the goal is to optimize graph structure given a process of interest, or to optimize the outcome of the process itself under fixed graph structure. Finally, we discuss the common challenges facing the field and open research questions. In contrast with other surveys, the present work focuses on non-canonical graph problems for which performant algorithms are typically not known and Reinforcement Learning is able to provide efficient and effective solutions.

Machine Learning,Artificial Intelligence
Representation Learning on Graphs: Methods and Applications

William L. Hamilton,Rex Ying,Jure Leskovec

DOI: https://doi.org/10.48550/arXiv.1709.05584

2017-09-17

Social and Information Networks

Abstract:Machine learning on graphs is an important and ubiquitous task with applications ranging from drug design to friendship recommendation in social networks. The primary challenge in this domain is finding a way to represent, or encode, graph structure so that it can be easily exploited by machine learning models. Traditionally, machine learning approaches relied on user-defined heuristics to extract features encoding structural information about a graph (e.g., degree statistics or kernel functions). However, recent years have seen a surge in approaches that automatically learn to encode graph structure into low-dimensional embeddings, using techniques based on deep learning and nonlinear dimensionality reduction. Here we provide a conceptual review of key advancements in this area of representation learning on graphs, including matrix factorization-based methods, random-walk based algorithms, and graph neural networks. We review methods to embed individual nodes as well as approaches to embed entire (sub)graphs. In doing so, we develop a unified framework to describe these recent approaches, and we highlight a number of important applications and directions for future work.
A Reinforcement Learning Model for Influence Maximization in Social Networks.

Chao Wang,Yiming Liu,Xiaofeng Gao,Guihai Chen

DOI: https://doi.org/10.1007/978-3-030-73197-7_48

2021-01-01

Abstract:Social influence maximization problem has been widely studied by the industrial and theoretical researchers over the years. However, with the skyrocketing scale of networks and growing complexity of application scenarios, traditional approximation approaches suffer from weak approximation guarantees and bad empirical performances. What's more, they can't be applied to new users in dynamic network. To tackle those problems, we introduce a social influence maximization algorithm via graph embedding and reinforcement learning. Nodes are represented in the graph with their embedding, and then we formulate a reinforcement learning model where both the states and the actions can be represented with vectors in low dimensional space. Now we can deal with graphs under various scenarios and sizes, just by learning parameters for the deep neural network. Hence, our model can be applied to both large-scale and dynamic social networks. Extensive real-world experiments show that our model significantly outperforms baselines across various data sets, and the algorithm learned on small-scale graphs can be generalized to large-scale ones.
Optimization-Based Algebraic Multigrid Coarsening Using Reinforcement Learning

Ali Taghibakhshi,Scott MacLachlan,Luke Olson,Matthew West

DOI: https://doi.org/10.48550/arXiv.2106.01854

2022-01-04

Abstract:Large sparse linear systems of equations are ubiquitous in science and engineering, such as those arising from discretizations of partial differential equations. Algebraic multigrid (AMG) methods are one of the most common methods of solving such linear systems, with an extensive body of underlying mathematical theory. A system of linear equations defines a graph on the set of unknowns and each level of a multigrid solver requires the selection of an appropriate coarse graph along with restriction and interpolation operators that map to and from the coarse representation. The efficiency of the multigrid solver depends critically on this selection and many selection methods have been developed over the years. Recently, it has been demonstrated that it is possible to directly learn the AMG interpolation and restriction operators, given a coarse graph selection. In this paper, we consider the complementary problem of learning to coarsen graphs for a multigrid solver, a necessary step in developing fully learnable AMG methods. We propose a method using a reinforcement learning (RL) agent based on graph neural networks (GNNs), which can learn to perform graph coarsening on small planar training graphs and then be applied to unstructured large planar graphs, assuming bounded node degree. We demonstrate that this method can produce better coarse graphs than existing algorithms, even as the graph size increases and other properties of the graph are varied. We also propose an efficient inference procedure for performing graph coarsening that results in linear time complexity in graph size.

Machine Learning
Reinforcement Learning on Graphs: A Survey

Mingshuo Nie,Dongming Chen,Dongqi Wang

DOI: https://doi.org/10.1109/tetci.2022.3222545

2023-01-01

IEEE Transactions on Emerging Topics in Computational Intelligence

Abstract:Graph mining tasks arise from many different application domains, including social networks, biological networks, transportation, and E-commerce, which have been receiving great attention from the theoretical and algorithmic design communities in recent years, and there has been some pioneering work employing the research-rich Reinforcement Learning (RL) techniques to address graph mining tasks. However, these fusion works are dispersed in different research domains, which makes them difficult to compare. In this survey, we provide a comprehensive overview of these fusion works and generalize these works to Graph Reinforcement Learning (GRL) as a unified formulation. We further discuss the applications of GRL methods across various domains, and simultaneously propose the key challenges and advantages of integrating graph mining and RL methods. Furthermore, we propose important directions and challenges to be solved in the future. To our knowledge, this is the latest work on a comprehensive survey of GRL, this work provides a global view and a learning resource for scholars. Based on our review, we create a collection of papers for both interested scholars who want to enter this rapidly developing domain and experts who would like to compare GRL methods.
Global Reinforcement Learning in Neural Networks.

Xiaolong Ma,Konstantin K. Likharev

DOI: https://doi.org/10.1109/tnn.2006.888376

2007-01-01

IEEE Transactions on Neural Networks

Abstract:In this letter, we have found a more general formulation of the REward Increment = Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility (REINFORCE) learning principle first suggested by Williams. The new formulation has enabled us to apply the principle to global reinforcement learning in networks with various sources of randomness, and to suggest several simple local rules for such networks. Numerical simulations have shown that for simple classification and reinforcement learning tasks, at least one family of the new learning rules gives results comparable to those provided by the famous Rules A r-i and A r-p for the Boltzmann machines
Reinforcement Learning for Scalable Logic Optimization with Graph Neural Networks

Xavier Timoneda,Lukas Cavigelli

DOI: https://doi.org/10.48550/arXiv.2105.01755

2021-05-05

Abstract:Logic optimization is an NP-hard problem commonly approached through hand-engineered heuristics. We propose to combine graph convolutional networks with reinforcement learning and a novel, scalable node embedding method to learn which local transforms should be applied to the logic graph. We show that this method achieves a similar size reduction as ABC on smaller circuits and outperforms it by 1.5-1.75x on larger random graphs.

Machine Learning
Goal-directed graph construction using reinforcement learning

Victor-Alexandru Darvariu,Stephen Hailes,Mirco Musolesi

DOI: https://doi.org/10.1098/rspa.2021.0168

2021-10-01

Abstract:Graphs can be used to represent and reason about systems and a variety of metrics have been devised to quantify their global characteristics. However, little is currently known about how to construct a graph or improve an existing one given a target objective. In this work, we formulate the construction of a graph as a decision-making process in which a central agent creates topologies by trial and error and receives rewards proportional to the value of the target objective. By means of this conceptual framework, we propose an algorithm based on reinforcement learning and graph neural networks to learn graph construction and improvement strategies. Our core case study focuses on robustness to failures and attacks, a property relevant for the infrastructure and communication networks that power modern society. Experiments on synthetic and real-world graphs show that this approach can outperform existing methods while being cheaper to evaluate. It also allows generalization to out-of-sample graphs, as well as to larger out-of-distribution graphs in some cases. The approach is applicable to the optimization of other global structural properties of graphs.
Graph Reinforcement Learning for Power Grids: A Comprehensive Survey

Mohamed Hassouna,Clara Holzhüter,Pawel Lytaev,Josephine Thomas,Bernhard Sick,Christoph Scholz

2024-08-26

Abstract:The rise of renewable energy and distributed generation requires new approaches to overcome the limitations of traditional methods. In this context, Graph Neural Networks are promising due to their ability to learn from graph-structured data. Combined with Reinforcement Learning, they can serve as control approaches to determine remedial network actions. This review analyses how Graph Reinforcement Learning (GRL) can improve representation learning and decision making in power grid use cases. Although GRL has demonstrated adaptability to unpredictable events and noisy data, it is primarily at a proof-of-concept stage. We highlight open challenges and limitations with respect to real-world applications.

Machine Learning
Explore with Dynamic Map: Graph Structured Reinforcement Learning

Jiarui Jin,Sijin Zhou,Weinan Zhang,Rasool Fakoor,David Wipf,Tong He,Yong Yu,Zheng Zhang,Alex Smola

2021-01-01

Abstract:In reinforcement learning, a map with states and transitions built based on historical trajectories is often helpful in exploration and exploitation. Even so, learning and planning on such a map within a sparse environment remains a challenge. As a step towards this goal, we propose Graph Structured Reinforcement Learning (GSRL), which utilizes historical trajectories to slowly adjust exploration directions and learn related experiences while rapidly updating the value function estimation. GSRL constructs a dynamic graph on top of state transitions in the replay buffer based on historical trajectories, and develops an attention strategy on the map to select an appropriate goal direction, which decomposes the task of reaching a distant goal state into a sequence of easier tasks. We also leverage graph structure to sample related trajectories for efficient value learning. Results demonstrate that GSRL can outperform the state-of-the-art algorithms in terms of sample efficiency on benchmarks with sparse reward functions.

Reinforcement learning for graph theory, I. Reimplementation of Wagner's approach

Reinforcement learning for graph theory, II. Small Ramsey numbers

A Systematization of the Wagner Framework: Graph Theory Conjectures and Reinforcement Learning

Graph Representations for Reinforcement Learning

Graph learning-based generation of abstractions for reinforcement learning

Revisiting Random Walks for Learning on Graphs

Reinforcement Learning Discovers Efficient Decentralized Graph Path Search Strategies

Towards Generalizability of Multi-Agent Reinforcement Learning in Graphs with Recurrent Message Passing

Deep Reinforcement Learning of Graph Matching.

Graph Reinforcement Learning for Network Control via Bi-Level Optimization

Reinforced Continual Learning for Graphs

Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective

Representation Learning on Graphs: Methods and Applications

A Reinforcement Learning Model for Influence Maximization in Social Networks.

Optimization-Based Algebraic Multigrid Coarsening Using Reinforcement Learning

Reinforcement Learning on Graphs: A Survey

Global Reinforcement Learning in Neural Networks.

Reinforcement Learning for Scalable Logic Optimization with Graph Neural Networks

Goal-directed graph construction using reinforcement learning

Graph Reinforcement Learning for Power Grids: A Comprehensive Survey

Explore with Dynamic Map: Graph Structured Reinforcement Learning