Large Language Models Can Learn Temporal Reasoning

Siheng Xiong,Ali Payani,Ramana Kompella,Faramarz Fekri

2024-06-11

Abstract:While large language models (LLMs) have demonstrated remarkable reasoning capabilities, they are not without their flaws and inaccuracies. Recent studies have introduced various methods to mitigate these limitations. Temporal reasoning (TR), in particular, presents a significant challenge for LLMs due to its reliance on diverse temporal concepts and intricate temporal logic. In this paper, we propose TG-LLM, a novel framework towards language-based TR. Instead of reasoning over the original context, we adopt a latent representation, temporal graph (TG) that enhances the learning of TR. A synthetic dataset (TGQA), which is fully controllable and requires minimal supervision, is constructed for fine-tuning LLMs on this text-to-TG translation task. We confirmed in experiments that the capability of TG translation learned on our dataset can be transferred to other TR tasks and benchmarks. On top of that, we teach LLM to perform deliberate reasoning over the TGs via Chain-of-Thought (CoT) bootstrapping and graph data augmentation. We observed that those strategies, which maintain a balance between usefulness and diversity, bring more reliable CoTs and final results than the vanilla CoT distillation.

Computation and Language

What problem does this paper attempt to address?

This paper mainly discusses the limitations of large language models (LLMs) in temporal reasoning (TR), and proposes a new framework called TG-LLM to improve the performance of LLMs in this field. Temporal reasoning involves understanding complex concepts such as event sequence, duration, and causality. In the paper, the authors create a synthetic dataset (TGQA) to train LLMs to convert text into temporal graphs (TG) and then reason on these graphs. They enable LLMs to generate reliable and consistent reasoning paths through chain-of-thought (CoT) guidance and graph data augmentation strategies. Specifically, TG-LLM consists of two steps: 1) translation from text to temporal graph, converting the original context into a temporal graph; 2) temporal graph reasoning, LLM performs chain-of-thought reasoning on the temporal graph. Experimental results show that the learned ability to transform text into temporal graphs from the TGQA dataset can be transferred to other TR tasks and benchmark tests. Moreover, the mentioned methods improve the reliability and consistency of the final reasoning results and better balance usefulness and diversity compared to the direct CoT distillation method. The paper also describes the process of creating the TGQA dataset, which is a fully controllable and minimally supervised dataset for text-to-temporal-graph alignment. By fine-tuning the LLM, they achieve better performance on other TR tasks and benchmark tests. Finally, the paper proposes two strategies, including CoT guidance and graph data augmentation, to teach LLMs to generate consistent and reliable chain-of-thought paths, thereby improving performance.

Large Language Models Can Learn Temporal Reasoning

Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

LTLBench: Towards Benchmarks for Evaluating Temporal Logic Reasoning in Large Language Models

Toward Adaptive Reasoning in Large Language Models with Thought Rollback

Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?

Unlocking Temporal Question Answering for Large Language Models Using Code Execution

Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models

Large Language Models can Learn Rules

Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models

On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning

Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives

Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought

Reasoning with Large Language Models, a Survey

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding

Are Large Language Models Temporally Grounded?

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph

Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning