An Open-source End-to-End Logic Optimization Framework for Large-scale Boolean Network with Reinforcement Learning

Zhen Li,Kaixiang Zhu,Xuegong Zhou,Lingli Wang

2024-03-26

Abstract:We propose an open-source end-to-end logic optimization framework for large-scale boolean network with reinforcement learning.

Artificial Intelligence

What problem does this paper attempt to address?

Reinforcement Learning for Scalable Logic Optimization with Graph Neural Networks

Xavier Timoneda,Lukas Cavigelli

DOI: https://doi.org/10.48550/arXiv.2105.01755

2021-05-05

Abstract:Logic optimization is an NP-hard problem commonly approached through hand-engineered heuristics. We propose to combine graph convolutional networks with reinforcement learning and a novel, scalable node embedding method to learn which local transforms should be applied to the logic graph. We show that this method achieves a similar size reduction as ABC on smaller circuits and outperforms it by 1.5-1.75x on larger random graphs.

Machine Learning
Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic

Xufeng Zhao,Mengdi Li,Wenhao Lu,Cornelius Weber,Jae Hee Lee,Kun Chu,Stefan Wermter

2024-03-26

Abstract:Recent advancements in large language models have showcased their remarkable generalizability across various domains. However, their reasoning abilities still have significant room for improvement, especially when confronted with scenarios requiring multi-step reasoning. Although large language models possess extensive knowledge, their reasoning often fails to effectively utilize this knowledge to establish a coherent thinking paradigm. These models sometimes show hallucinations as their reasoning procedures are unconstrained by logical principles. Aiming at improving the zero-shot chain-of-thought reasoning ability of large language models, we propose LoT (Logical Thoughts), a self-improvement prompting framework that leverages principles rooted in symbolic logic, particularly Reductio ad Absurdum, to systematically verify and rectify the reasoning processes step by step. Experimental evaluations conducted on language tasks in diverse domains, including arithmetic, commonsense, symbolic, causal inference, and social problems, demonstrate the efficacy of enhanced reasoning by logic. The implementation code for LoT can be accessed at: <a class="link-external link-https" href="https://github.com/xf-zhao/LoT" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Machine Learning,Symbolic Computation
OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

Jun Wang,Meng Fang,Ziyu Wan,Muning Wen,Jiachen Zhu,Anjie Liu,Ziqin Gong,Yan Song,Lei Chen,Lionel M. Ni,Linyi Yang,Ying Wen,Weinan Zhang

2024-10-13

Abstract:In this technical report, we introduce OpenR, an open-source framework designed to integrate key components for enhancing the reasoning capabilities of large language models (LLMs). OpenR unifies data acquisition, reinforcement learning training (both online and offline), and non-autoregressive decoding into a cohesive software platform. Our goal is to establish an open-source platform and community to accelerate the development of LLM reasoning. Inspired by the success of OpenAI's o1 model, which demonstrated improved reasoning abilities through step-by-step reasoning and reinforcement learning, OpenR integrates test-time compute, reinforcement learning, and process supervision to improve reasoning in LLMs. Our work is the first to provide an open-source framework that explores the core techniques of OpenAI's o1 model with reinforcement learning, achieving advanced reasoning capabilities beyond traditional autoregressive methods. We demonstrate the efficacy of OpenR by evaluating it on the MATH dataset, utilising publicly available data and search methods. Our initial experiments confirm substantial gains, with relative improvements in reasoning and performance driven by test-time computation and reinforcement learning through process reward models. The OpenR framework, including code, models, and datasets, is accessible at <a class="link-external link-https" href="https://openreasoner.github.io" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence,Computation and Language
An Open-Source AMS Circuit Optimization Framework Based on Reinforcement Learning—From Specifications to Layouts

Zonghao Li,Anthony Chan Carusone

DOI: https://doi.org/10.1109/access.2024.3478832

IF: 3.9

2024-10-19

IEEE Access

Abstract:This paper presents a fully open-sourced AMS integrated circuit optimization framework based on reinforcement learning (RL). Specifically, given a certain circuit topology and target specifications, this framework optimizes the circuit in both schematic and post-layout phases. We propose using the heterogeneous graph neural network as the function approximator for RL. Optimization results suggest that it can achieve higher reward values with fewer iterations than the homogeneous graph neural networks. We demonstrate the applications of transfer learning (TL) in optimizing circuits in a different technology node. Furthermore, we show that by transferring the knowledge of schematic-level optimization, the trained RL agent can optimize the post-layout performance more efficiently than optimizing post-layout performance from scratch. To showcase the workflow of our approach, we extended our prior work to optimize latched comparators in the SKY130 and GF180MCU processes. Simulation results demonstrate that our framework can satisfy various target specifications and generate LVS/DRC clean circuit layouts.

computer science, information systems,telecommunications,engineering, electrical & electronic
An Efficient Reinforcement Learning Based Framework for Exploring Logic Synthesis

Yu Qian,Xuegong Zhou,Hao Zhou,Lingli Wang

DOI: https://doi.org/10.1145/3632174

IF: 1.447

2023-11-10

ACM Transactions on Design Automation of Electronic Systems

Abstract:Logic synthesis is a crucial step in electronic design automation tools. The rapid developments of reinforcement learning (RL) have enabled the automated exploration of logic synthesis. Existing RL based methods may lead to data inefficiency, and the exploration approaches for FPGA and ASIC technology mapping in recent works lack the flexibility of the learning process. This work proposes ESE, a reinforcement learning based framework to efficiently learn the logic synthesis process. The framework supports the modeling of logic optimization and technology mapping for FPGA and ASIC. The optimization for the execution time of the synthesis script is also considered. For the modeling of FPGA mapping, the logic optimization and technology mapping are combined to be learned in a flexible way. For the modeling of ASIC mapping, the standard cell based optimization and LUT optimization operations are incorporated into the ASIC synthesis flow. To improve the utilization of samples, the Proximal Policy Optimization model is adopted. Furthermore, the framework is enhanced by supporting MIG based synthesis exploration. Experiments show that for FPGA technology mapping on the VTR benchmark, the average LUT-Level-Product and script runtime are improved by more than 18.3% and 12.4% respectively than previous works. For ASIC mapping on the EPFL benchmark, the average Area-Delay-Product is improved by 14.5%.

computer science, software engineering, hardware & architecture
Efficient Reinforcement Learning Framework for Automated Logic Synthesis Exploration

Yu Qian,Xuegong Zhou,Hao Zhou,Lingli Wang

DOI: https://doi.org/10.1109/ICFPT56656.2022.9974330

2022-01-01

Abstract:Logic synthesis is a crucial step in electronic design automation tools for integrated circuit design. In recent years, the development of reinforcement learning (RL) has enabled the designers to automatically explore the logic synthesis process. Existing RL based methods typically use conventional on-policy models, which leads to data inefficiency. Moreover, the exploration approach for FPGA technology mapping in recent works lacks the flexibility of the learning process. In this work, we propose ESE, a reinforcement learning based framework to efficiently learn the logic synthesis process. The framework supports the modeling for both the logic optimization and the FPGA technology mapping. The reward functions and terminal conditions in the RL environment are designed to efficiently guide the optimization of the metrics and execution time. For the modeling of FPGA mapping, the logic optimization and technology mapping are combined to be learned in a flexible way. Moreover, the Proximal Policy Optimization model is adopted to improve the utilization of samples. The proposed framework is evaluated on several common benchmarks. For the logic optimization on the EPFL benchmark, compared with previous works, the proposed method obtains an 11.3% improvement in the average quality (node-level-product) and reduces the execution time by 13.7%. For the FPGA technology mapping on the VTR benchmark, our method improves the average quality (LUT-level-product) by 14.8%, and reduces the execution time by 14.4% compared with the recent work.
Neuro-symbolic Learning Yielding Logical Constraints

Zenan Li,Yunpeng Huang,Zhaoyu Li,Yuan Yao,Jingwei Xu,Taolue Chen,Xiaoxing Ma,Jian Lu

2024-10-28

Abstract:Neuro-symbolic systems combine the abilities of neural perception and logical reasoning. However, end-to-end learning of neuro-symbolic systems is still an unsolved challenge. This paper proposes a natural framework that fuses neural network training, symbol grounding, and logical constraint synthesis into a coherent and efficient end-to-end learning process. The capability of this framework comes from the improved interactions between the neural and the symbolic parts of the system in both the training and inference stages. Technically, to bridge the gap between the continuous neural network and the discrete logical constraint, we introduce a difference-of-convex programming technique to relax the logical constraints while maintaining their precision. We also employ cardinality constraints as the language for logical constraint learning and incorporate a trust region method to avoid the degeneracy of logical constraint in learning. Both theoretical analyses and empirical evaluations substantiate the effectiveness of the proposed framework.

Artificial Intelligence,Machine Learning
ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling

Chenyu Huang,Zhengyang Tang,Dongdong Ge,Shixi Hu,Ruoqing Jiang,Benyou Wang,Zizhuo Wang,Xin Zheng

2024-11-15

Abstract:Optimization modeling and solving play a critical role in the application of Operations Research (OR) tools to address real-world problems, yet they pose challenges and require extensive expertise from OR experts. With the advent of large language models (LLMs), new opportunities have emerged to streamline and automate these tasks. However, current research predominantly relies on closed-source LLMs such as GPT-4, along with extensive prompt engineering techniques. This reliance stems from the scarcity of high-quality training datasets for optimization modeling, resulting in elevated costs, prolonged processing times, and privacy concerns. To address these challenges, our work is the first to propose a viable path for training open-source LLMs that are capable of optimization modeling as well as developing and executing solver codes, eventually leading to a superior ability for automating optimization modeling and solving. Particularly, we introduce a semi-automated data synthesis framework designed for optimization modeling issues, named OR-Instruct. This framework merges the training data requirements of large models with the unique characteristics of optimization modeling problems, and allows for customizable enhancements tailored to specific scenarios or modeling types. To evaluate the performance of our proposed framework, we present the IndustryOR benchmark, the inaugural industrial standard for evaluating LLMs in solving practical OR problems. Utilizing data synthesized through OR-Instruct, we train various open-source LLMs with a capacity of 7 billion parameters (dubbed ORLMs). The resulting model demonstrates significantly enhanced optimization modeling capabilities, achieving state-of-the-art performance across the NL4OPT, MAMO, and IndustryOR benchmarks. Our code and data are available at \url{<a class="link-external link-https" href="https://github.com/Cardinal-Operations/ORLM" rel="external noopener nofollow">this https URL</a>}.

Computation and Language,Artificial Intelligence,Computational Engineering, Finance, and Science,Machine Learning
Logic Synthesis Optimization Sequence tuning using RL-based LSTM and Graph Isomorphism Network

Chenghao Yang,Yinshui Xia,Zhufei Chu,Xiaojing Zha

DOI: https://doi.org/10.1109/tcsii.2022.3168344

2022-01-01

Abstract:As a key step in the IC design flow, logic synthesis involves various logic optimization algorithms to be iteratively applied to the circuit. However, how these algorithms are used is usually determined by heuristics, and it does not always yield well optimizations on all circuits. To achieve well optimized results, engineers need to tune the sequence consisting of these logic optimization algorithms based on their knowledge. To overcome this limitation, in this brief, reinforcement learning (RL) proximal policy optimization (PPO) is proposed to train an agent to tune the optimization sequence. Specifically, graph isomorphic network with edge feature aggregation capability (GINE) is used to learn circuit representations and use circuit representations as state representations for the reinforcement learning agent. Furthermore, to enable the agent learning from historical operations, the Long Short-Term Memory (LSTM) is further embedded in reinforcement learning. The evaluation of the EPFL arithmetic benchmark shows that our model improves the area optimization under the delay constraint by 21.21% over existing methods.
BoolGebra: Attributed Graph-learning for Boolean Algebraic Manipulation

Yingjie Li,Anthony Agnesina,Yanqing Zhang,Haoxing Ren,Cunxi Yu

2024-01-19

Abstract:Boolean algebraic manipulation is at the core of logic synthesis in Electronic Design Automation (EDA) design flow. Existing methods struggle to fully exploit optimization opportunities, and often suffer from an explosive search space and limited scalability efficiency. This work presents BoolGebra, a novel attributed graph-learning approach for Boolean algebraic manipulation that aims to improve fundamental logic synthesis. BoolGebra incorporates Graph Neural Networks (GNNs) and takes initial feature embeddings from both structural and functional information as inputs. A fully connected neural network is employed as the predictor for direct optimization result predictions, significantly reducing the search space and efficiently locating the optimization space. The experiments involve training the BoolGebra model w.r.t design-specific and cross-design inferences using the trained model, where BoolGebra demonstrates generalizability for cross-design inference and its potential to scale from small, simple training datasets to large, complex inference datasets. Finally, BoolGebra is integrated with existing synthesis tool ABC to perform end-to-end logic minimization evaluation w.r.t SOTA baselines.

Hardware Architecture,Artificial Intelligence,Machine Learning
Deep Reinforcement Learning Based Optimal Infinite-Horizon Control of Probabilistic Boolean Control Networks

Jingjie Ni,Fangfei Li,Zheng-Guang Wu

2023-04-07

Abstract:In this paper, a deep reinforcement learning based method is proposed to obtain optimal policies for optimal infinite-horizon control of probabilistic Boolean control networks (PBCNs). Compared with the existing literatures, the proposed method is model-free, namely, the system model and the initial states needn't to be known. Meanwhile, it is suitable for large-scale PBCNs. First, we establish the connection between deep reinforcement learning and optimal infinite-horizon control, and structure the problem into the framework of the Markov decision process. Then, PBCNs are defined as large-scale or small-scale, depending on whether the memory of the action-values exceeds the RAM of the computer. Based on the newly introduced definition, Q-learning (QL) and double deep Q-network (DDQN) are applied to the optimal infinite-horizon control of small-scale and large-scale PBCNs, respectively. Meanwhile, the optimal state feedback controllers are designed. Finally, two examples are presented, which are a small-scale PBCN with 3 nodes, and a large-scale one with 28 nodes. To verify the convergence of QL and DDQN, the optimal control policy and the optimal action-values, which are obtained from both the algorithms, are compared with the ones based on a model-based method named policy iteration. Meanwhile, the performance of QL is compared with DDQN in the small-scale PBCN.

Systems and Control
OpenLS-DGF: An Adaptive Open-Source Dataset Generation Framework for Machine Learning Tasks in Logic Synthesis

Liwei Ni,Rui Wang,Miao Liu,Xingyu Meng,Xiaoze Lin,Junfeng Liu,Guojie Luo,Zhufei Chu,Weikang Qian,Xiaoyan Yang,Biwei Xie,Xingquan Li,Huawei Li

2024-11-16

Abstract:This paper introduces OpenLS-DGF, an adaptive logic synthesis dataset generation framework, to enhance machine learning~(ML) applications within the logic synthesis process. Previous dataset generation flows were tailored for specific tasks or lacked integrated machine learning capabilities. While OpenLS-DGF supports various machine learning tasks by encapsulating the three fundamental steps of logic synthesis: Boolean representation, logic optimization, and technology mapping. It preserves the original information in both Verilog and machine-learning-friendly GraphML formats. The verilog files offer semi-customizable capabilities, enabling researchers to insert additional steps and incrementally refine the generated dataset. Furthermore, OpenLS-DGF includes an adaptive circuit engine that facilitates the final dataset management and downstream tasks. The generated OpenLS-D-v1 dataset comprises 46 combinational designs from established benchmarks, totaling over 966,000 Boolean circuits. OpenLS-D-v1 supports integrating new data features, making it more versatile for new challenges. This paper demonstrates the versatility of OpenLS-D-v1 through four distinct downstream tasks: circuit classification, circuit ranking, quality of results (QoR) prediction, and probability prediction. Each task is chosen to represent essential steps of logic synthesis, and the experimental results show the generated dataset from OpenLS-DGF achieves prominent diversity and applicability. The source code and datasets are available at <a class="link-external link-https" href="https://github.com/Logic-Factory/ACE/blob/master/OpenLS-DGF/readme.md" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence
Graph Reinforcement Learning for Network Control via Bi-Level Optimization

Daniele Gammelli,James Harrison,Kaidi Yang,Marco Pavone,Filipe Rodrigues,Francisco C. Pereira

2023-05-16

Abstract:Optimization problems over dynamic networks have been extensively studied and widely used in the past decades to formulate numerous real-world problems. However, (1) traditional optimization-based approaches do not scale to large networks, and (2) the design of good heuristics or approximation algorithms often requires significant manual trial-and-error. In this work, we argue that data-driven strategies can automate this process and learn efficient algorithms without compromising optimality. To do so, we present network control problems through the lens of reinforcement learning and propose a graph network-based framework to handle a broad class of problems. Instead of naively computing actions over high-dimensional graph elements, e.g., edges, we propose a bi-level formulation where we (1) specify a desired next state via RL, and (2) solve a convex program to best achieve it, leading to drastically improved scalability and performance. We further highlight a collection of desirable features to system designers, investigate design decisions, and present experiments on real-world control problems showing the utility, scalability, and flexibility of our framework.

Machine Learning,Systems and Control,Optimization and Control
EasySO: Exploration-enhanced Reinforcement Learning for Logic Synthesis Sequence Optimization and a Comprehensive RL Environment

Jianyong Yuan,Peiyu Wang,Junjie Ye,Mingxuan Yuan,Jianye Hao,Junchi Yan

DOI: https://doi.org/10.1109/ICCAD57390.2023.10323973

2023-01-01

Abstract:Optimizing the quality of results (QoR) of a circuit during the logic synthesis (LS) phase in chip design is critical yet challenging. While most existing methods often mitigate the computational hardness by restricting the action space to a small set of operators and fixing the operator's parameters, they are susceptible to local minima and may not meet the high demand from industrial cases. In this paper, we develop a more comprehensive optimization approach via sample-efficient reinforcement learning (RL). Specifically, we first build a complete logic synthesis-RL environment, where the action space consists of three types of operators: logic optimization, technology mapping, and post-mapping, along with their associated continuous/binary parameters for optimization as well. Based on this environment, we devise a hybrid proximal policy optimization (PPO) model to handle both discrete operators and parameters and design a distributed architecture to improve sample collection efficiency. Furthermore, we devise a dynamic exploration module to improve the exploration efficiency under the constraint of limited samples. We term our method as Exploration-enhanced RL for Logic Synthesis Sequence Optimization(EasySO). Results on the EPFL benchmark show that our method significantly outperforms current state-of-the-art models based on Bayesian optimization (BO) and the previous RL-based methods. Compared to resyn2, our EasySO achieves an average of 25.4% LUT-6 count optimization without sacrificing level values. Moreover, as of the time for this submission, we rank 26 first places among 40 optimization targets in the EPFL competition.
DeepLogic: Joint Learning of Neural Perception and Logical Reasoning.

Xuguang Duan,Xin Wang,Peilin Zhao,Guangyao Shen,Wenwu Zhu

DOI: https://doi.org/10.1109/tpami.2022.3191093

IF: 23.6

2022-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Neural-symbolic learning, aiming to combine the perceiving power of neural perception and the reasoning power of symbolic logic together, has drawn increasing research attention. However, existing works simply cascade the two components together and optimize them isolatedly, failing to utilize the mutual enhancing information between them. To address this problem, we propose DeepLogic, a framework with joint learning of neural perception and logical reasoning, such that these two components are jointly optimized through mutual supervision signals. In particular, the proposed DeepLogic framework contains a deep-logic module that is capable of representing complex first-order-logic formulas in a tree structure with basic logic operators. We then theoretically quantify the mutual supervision signals and propose the deep&logic optimization algorithm for joint optimization. We further prove the convergence of DeepLogic and conduct extensive experiments on model performance, convergence, and generalization, as well as its extension to the continuous domain. The experimental results show that through jointly learning both perceptual ability and logic formulas in a weakly supervised manner, our proposed DeepLogic framework can significantly outperform DNN-based baselines by a great margin and beat other strong baselines without out-of-box tools.
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

Liangming Pan,Alon Albalak,Xinyi Wang,William Yang Wang

2023-10-19

Abstract:Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at <a class="link-external link-https" href="https://github.com/teacherpeterpan/Logic-LLM" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence
Logic Synthesis Optimization with Predictive Self-Supervision via Causal Transformers

Raika Karimi,Faezeh Faez,Yingxue Zhang,Xing Li,Lei Chen,Mingxuan Yuan,Mahdi Biparva

2024-09-17

Abstract:Contemporary hardware design benefits from the abstraction provided by high-level logic gates, streamlining the implementation of logic circuits. Logic Synthesis Optimization (LSO) operates at one level of abstraction within the Electronic Design Automation (EDA) workflow, targeting improvements in logic circuits with respect to performance metrics such as size and speed in the final layout. Recent trends in the field show a growing interest in leveraging Machine Learning (ML) for EDA, notably through ML-guided logic synthesis utilizing policy-based Reinforcement Learning (RL) methods.Despite these advancements, existing models face challenges such as overfitting and limited generalization, attributed to constrained public circuits and the expressiveness limitations of graph encoders. To address these hurdles, and tackle data scarcity issues, we introduce LSOformer, a novel approach harnessing Autoregressive transformer models and predictive SSL to predict the trajectory of Quality of Results (QoR). LSOformer integrates cross-attention modules to merge insights from circuit graphs and optimization sequences, thereby enhancing prediction accuracy for QoR metrics. Experimental studies validate the effectiveness of LSOformer, showcasing its superior performance over baseline architectures in QoR prediction tasks, where it achieves improvements of 5.74%, 4.35%, and 17.06% on the EPFL, OABCD, and proprietary circuits datasets, respectively, in inductive setup.

Artificial Intelligence,Machine Learning
LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning

Jin Jiang,Yuchen Yan,Yang Liu,Yonggang Jin,Shuai Peng,Mengdi Zhang,Xunliang Cai,Yixin Cao,Liangcai Gao,Zhi Tang

2024-09-20

Abstract:In this paper, we present a novel approach, called LogicPro, to enhance Large Language Models (LLMs) complex Logical reasoning through Program Examples. We do this effectively by simply utilizing widely available algorithmic problems and their code solutions. First, we constructed diverse test samples input based on algorithmic questions and code solutions. Then, we designed different complex reasoning questions based on algorithmic problems and test samples. Finally, combining the intermediate variable outputs of the code solutions and the complex reasoning questions, we derived the reasoning process and the final answer. With this approach, we can construct a dataset that is sufficiently difficult (all models are ineffective), diverse (synthesized from 2,360 different algorithmic questions), and scalable (building different test samples and collecting more algorithmic questions). In addition, we obtain a high-quality reasoning process guided by the values of intermediate variables. As a result, our approach achieves significant improvements in multiple models for the BBH$^{27}$, GSM8K, HellSwag, Logicqa, Reclor, and RTE datasets, outperforming a wide range of existing reasoning datasets.

Computation and Language
RLOR: A Flexible Framework of Deep Reinforcement Learning for Operation Research

Ching Pui Wan,Tung Li,Jason Min Wang

2023-03-23

Abstract:Reinforcement learning has been applied in operation research and has shown promise in solving large combinatorial optimization problems. However, existing works focus on developing neural network architectures for certain problems. These works lack the flexibility to incorporate recent advances in reinforcement learning, as well as the flexibility of customizing model architectures for operation research problems. In this work, we analyze the end-to-end autoregressive models for vehicle routing problems and show that these models can benefit from the recent advances in reinforcement learning with a careful re-implementation of the model architecture. In particular, we re-implemented the Attention Model and trained it with Proximal Policy Optimization (PPO) in CleanRL, showing at least 8 times speed up in training time. We hereby introduce RLOR, a flexible framework for Deep Reinforcement Learning for Operation Research. We believe that a flexible framework is key to developing deep reinforcement learning models for operation research problems. The code of our work is publicly available at <a class="link-external link-https" href="https://github.com/cpwan/RLOR" rel="external noopener nofollow">this https URL</a>.

Optimization and Control,Machine Learning,Neural and Evolutionary Computing
Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

Tongxuan Liu,Wenjiang Xu,Weizhe Huang,Xingyu Wang,Jiaxing Wang,Hailong Yang,Jing Li

2024-09-26

Abstract:Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information from input context, and utilizes the generated logical information as an additional augmentation to the input prompts, thereby enhancing the capability of logical reasoning. The LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, the LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%; moreover, it improves Chain-of-Thought with Self-Consistency's performance on LogiQA by +5%; additionally, it boosts performance of Tree-of-Thoughts on ProofWriter dataset by +8%.

Computation and Language

An Open-source End-to-End Logic Optimization Framework for Large-scale Boolean Network with Reinforcement Learning

Reinforcement Learning for Scalable Logic Optimization with Graph Neural Networks

Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic

OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

An Open-Source AMS Circuit Optimization Framework Based on Reinforcement Learning—From Specifications to Layouts

An Efficient Reinforcement Learning Based Framework for Exploring Logic Synthesis

Efficient Reinforcement Learning Framework for Automated Logic Synthesis Exploration

Neuro-symbolic Learning Yielding Logical Constraints

ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling

Logic Synthesis Optimization Sequence tuning using RL-based LSTM and Graph Isomorphism Network

BoolGebra: Attributed Graph-learning for Boolean Algebraic Manipulation

Deep Reinforcement Learning Based Optimal Infinite-Horizon Control of Probabilistic Boolean Control Networks

OpenLS-DGF: An Adaptive Open-Source Dataset Generation Framework for Machine Learning Tasks in Logic Synthesis

Graph Reinforcement Learning for Network Control via Bi-Level Optimization

EasySO: Exploration-enhanced Reinforcement Learning for Logic Synthesis Sequence Optimization and a Comprehensive RL Environment

DeepLogic: Joint Learning of Neural Perception and Logical Reasoning.

Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

Logic Synthesis Optimization with Predictive Self-Supervision via Causal Transformers

LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning

RLOR: A Flexible Framework of Deep Reinforcement Learning for Operation Research

Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models