A corpus and cloze evaluation for deeper understanding of commonsense stories

Mostafazadeh Nasrin, Chambers Nathanael, He Xiaodong, Parikh Devi, Batra Dhruv, Vanderwende Lucy, Allen James

2016-01-01

Abstract:

What problem does this paper attempt to address?

A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories

Nasrin Mostafazadeh,Nathanael Chambers,Xiaodong He,Devi Parikh,Dhruv Batra,Lucy Vanderwende,Pushmeet Kohli,James Allen

DOI: https://doi.org/10.48550/arXiv.1604.01696

2016-04-06

Computation and Language

Abstract:Representation and learning of commonsense knowledge is one of the foundational problems in the quest to enable deep language understanding. This issue is particularly challenging for understanding casual and correlational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the 'Story Cloze Test'. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of ~50k five-sentence commonsense stories, ROCStories, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation. Experimental evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learning, and offer suggestions for deeper language understanding.
Building a commonsense knowledge base for context-awareness inference

Li Zhang,Shijian Li,Gang Pan

DOI: https://doi.org/10.1109/ICCA.2013.6565173

2013-01-01

Abstract:Current context-aware systems often model rigid inference rules for limited user contexts, which constrain their effectiveness in real world usage. In order to achieve flexible context-aware inference, a commonsense knowledge base is essential. But such knowledge base is hard to construct manually. This paper proposes some automatic algorithms to extract commonsense directly from text corpuses. We evaluate the extraction algorithms with comparison with human annotators. We also evaluate the effectiveness of the knowledge base in practical context-awareness inference.
Incorporating Structured Commonsense Knowledge in Story Completion

Jiaao Chen,Jianshu Chen,Zhou Yu

DOI: https://doi.org/10.48550/arXiv.1811.00625

2018-11-02

Abstract:The ability to select an appropriate story ending is the first step towards perfect narrative comprehension. Story ending prediction requires not only the explicit clues within the context, but also the implicit knowledge (such as commonsense) to construct a reasonable and consistent story. However, most previous approaches do not explicitly use background commonsense knowledge. We present a neural story ending selection model that integrates three types of information: narrative sequence, sentiment evolution and commonsense knowledge. Experiments show that our model outperforms state-of-the-art approaches on a public dataset, ROCStory Cloze Task , and the performance gain from adding the additional commonsense knowledge is significant.

Computation and Language
Paragraph-level Commonsense Transformers with Recurrent Memory

Saadia Gabriel,Chandra Bhagavatula,Vered Shwartz,Ronan Le Bras,Maxwell Forbes,Yejin Choi

DOI: https://doi.org/10.48550/arXiv.2010.01486

2021-02-02

Abstract:Human understanding of narrative texts requires making commonsense inferences beyond what is stated explicitly in the text. A recent model, COMET, can generate such implicit commonsense inferences along several dimensions such as pre- and post-conditions, motivations, and mental states of the participants. However, COMET was trained on commonsense inferences of short phrases, and is therefore discourse-agnostic. When presented with each sentence of a multi-sentence narrative, it might generate inferences that are inconsistent with the rest of the narrative. We present the task of discourse-aware commonsense inference. Given a sentence within a narrative, the goal is to generate commonsense inferences along predefined dimensions, while maintaining coherence with the rest of the narrative. Such large-scale paragraph-level annotation is hard to get and costly, so we use available sentence-level annotations to efficiently and automatically construct a distantly supervised corpus. Using this corpus, we train PARA-COMET, a discourse-aware model that incorporates paragraph-level information to generate coherent commonsense inferences from narratives. PARA-COMET captures both semantic knowledge pertaining to prior world knowledge, and episodic knowledge involving how current events relate to prior and future events in a narrative. Our results show that PARA-COMET outperforms the sentence-level baselines, particularly in generating inferences that are both coherent and novel.

Computation and Language,Machine Learning
A Corpus for Understanding and Generating Moral Stories

Jian Guan,Ziqi Liu,Minlie Huang

DOI: https://doi.org/10.48550/arXiv.2204.09438

2022-04-20

Abstract:Teaching morals is one of the most important purposes of storytelling. An essential ability for understanding and writing moral stories is bridging story plots and implied morals. Its challenges mainly lie in: (1) grasping knowledge about abstract concepts in morals, (2) capturing inter-event discourse relations in stories, and (3) aligning value preferences of stories and morals concerning good or bad behavior. In this paper, we propose two understanding tasks and two generation tasks to assess these abilities of machines. We present STORAL, a new dataset of Chinese and English human-written moral stories. We show the difficulty of the proposed tasks by testing various models with automatic and manual evaluation on STORAL. Furthermore, we present a retrieval-augmented algorithm that effectively exploits related concepts or events in training sets as additional guidance to improve performance on these tasks.

Computation and Language
Natural Language Processing with Commonsense Knowledge: A Survey

Yubo Xie,Zonghui Liu,Zongyang Ma,Fanyuan Meng,Yan Xiao,Fahui Miao,Pearl Pu

2024-09-13

Abstract:Commonsense knowledge is essential for advancing natural language processing (NLP) by enabling models to engage in human-like reasoning, which requires a deeper understanding of context and often involves making inferences based on implicit external knowledge. This paper explores the integration of commonsense knowledge into various NLP tasks. We begin by reviewing prominent commonsense knowledge bases and then discuss the benchmarks used to evaluate the commonsense reasoning capabilities of NLP models, particularly language models. Furthermore, we highlight key methodologies for incorporating commonsense knowledge and their applications across different NLP tasks. The paper also examines the challenges and emerging trends in enhancing NLP systems with commonsense reasoning. All literature referenced in this survey can be accessed via our GitHub repository: <a class="link-external link-https" href="https://github.com/yuboxie/awesome-commonsense" rel="external noopener nofollow">this https URL</a>.

Computation and Language
Event2Mind: Commonsense Inference on Events, Intents, and Reactions

Hannah Rashkin,Maarten Sap,Emily Allaway,Noah A. Smith,Yejin Choi

DOI: https://doi.org/10.48550/arXiv.1805.06939

2019-06-15

Abstract:We investigate a new commonsense inference task: given an event described in a short free-form text ("X drinks coffee in the morning"), a system reasons about the likely intents ("X wants to stay awake") and reactions ("X feels alert") of the event's participants. To support this study, we construct a new crowdsourced corpus of 25,000 event phrases covering a diverse range of everyday events and situations. We report baseline performance on this task, demonstrating that neural encoder-decoder models can successfully compose embedding representations of previously unseen events and reason about the likely intents and reactions of the event participants. In addition, we demonstrate how commonsense inference on people's intents and reactions can help unveil the implicit gender inequality prevalent in modern movie scripts.

Computation and Language
CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models

Dan Shi,Chaobin You,Jiantao Huang,Taihao Li,Deyi Xiong

2023-12-20

Abstract:As an indispensable ingredient of intelligence, commonsense reasoning is crucial for large language models (LLMs) in real-world scenarios. In this paper, we propose CORECODE, a dataset that contains abundant commonsense knowledge manually annotated on dyadic dialogues, to evaluate the commonsense reasoning and commonsense conflict detection capabilities of Chinese LLMs. We categorize commonsense knowledge in everyday conversations into three dimensions: entity, event, and social interaction. For easy and consistent annotation, we standardize the form of commonsense knowledge annotation in open-domain dialogues as "domain: slot = value". A total of 9 domains and 37 slots are defined to capture diverse commonsense knowledge. With these pre-defined domains and slots, we collect 76,787 commonsense knowledge annotations from 19,700 dialogues through crowdsourcing. To evaluate and enhance the commonsense reasoning capability for LLMs on the curated dataset, we establish a series of dialogue-level reasoning and detection tasks, including commonsense knowledge filling, commonsense knowledge generation, commonsense conflict phrase detection, domain identification, slot identification, and event causal inference. A wide variety of existing open-source Chinese LLMs are evaluated with these tasks on our dataset. Experimental results demonstrate that these models are not competent to predict CORECODE's plentiful reasoning content, and even ChatGPT could only achieve 0.275 and 0.084 accuracy on the domain identification and slot identification tasks under the zero-shot setting. We release the data and codes of CORECODE at <a class="link-external link-https" href="https://github.com/danshi777/CORECODE" rel="external noopener nofollow">this https URL</a> to promote commonsense reasoning evaluation and study of LLMs in the context of daily conversations.

Computation and Language
A Manual Experiment On Commonsense Knowledge Acquisition From Web Corpora

Yao Zhu,Liang-Jun Zang,Ya-Nan Cao,Dong-Sheng Wang,Cun-Gen Cao

DOI: https://doi.org/10.1109/ICMLC.2008.4620655

2008-01-01

Abstract:Acquiring commonsense knowledge from text is an important but challenging problem. In this paper, we described a three-subject experiment on commonsense knowledge acquisition from Chinese sentences extracted from a web corpus, aiming to investigate how people acquire commonsensical assertions from given sentences. We analyzed the experiment results from the perspectives of agreement test, concordance test, and divergence test. An important conclusion of our experiment is that sentences are different in their suitability, i.e. difficulty grade, for common sense knowledge acquisition. And this difficulty grade also affects the number of commonsensical assertions acquired from a sentence, as well as the difference among the acquisition performances of different human subjects. We also discussed the problem of characterizing the difficulty grade by co-occurrence frequency of words and basic level category words.
TransOMCS: From Linguistic Graphs to Commonsense Knowledge

Hongming Zhang,Daniel Khashabi,Yangqiu Song,Dan Roth

DOI: https://doi.org/10.48550/arXiv.2005.00206

2020-05-01

Abstract:Commonsense knowledge acquisition is a key problem for artificial intelligence. Conventional methods of acquiring commonsense knowledge generally require laborious and costly human annotations, which are not feasible on a large scale. In this paper, we explore a practical way of mining commonsense knowledge from linguistic graphs, with the goal of transferring cheap knowledge obtained with linguistic patterns into expensive commonsense knowledge. The result is a conversion of ASER [Zhang et al., 2020], a large-scale selectional preference knowledge resource, into TransOMCS, of the same representation as ConceptNet [Liu and Singh, 2004] but two orders of magnitude larger. Experimental results demonstrate the transferability of linguistic knowledge to commonsense knowledge and the effectiveness of the proposed approach in terms of quantity, novelty, and quality. TransOMCS is publicly available at: <a class="link-external link-https" href="https://github.com/HKUST-KnowComp/TransOMCS" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence,Computation and Language
UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations

Wenting Zhao,Justin T Chiu,Jena D. Hwang,Faeze Brahman,Jack Hessel,Sanjiban Choudhury,Yejin Choi,Xiang Lorraine Li,Alane Suhr

2024-05-01

Abstract:Language technologies that accurately model the dynamics of events must perform commonsense reasoning. Existing work evaluating commonsense reasoning focuses on making inferences about common, everyday situations. To instead investigate the ability to model unusual, unexpected, and unlikely situations, we explore the task of uncommonsense abductive reasoning. Given a piece of context with an unexpected outcome, this task requires reasoning abductively to generate an explanation that makes the unexpected outcome more likely in the context. To this end, we curate and release a new English language corpus called UNcommonsense. We characterize the performance differences between human explainers and the best-performing large language models, finding that model-enhanced human-written explanations achieve the highest quality by trading off between specificity and diversity. Finally, we experiment with several imitation learning algorithms to train open and accessible language models on this task. When compared with the vanilla supervised fine-tuning approach, these methods consistently reduce lose rates on both common and uncommonsense abductive reasoning judged by human evaluators.

Computation and Language
A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context

Takuma Udagawa,Akiko Aizawa

DOI: https://doi.org/10.48550/arXiv.1907.03399

2019-07-08

Abstract:Common grounding is the process of creating, repairing and updating mutual understandings, which is a critical aspect of sophisticated human communication. However, traditional dialogue systems have limited capability of establishing common ground, and we also lack task formulations which introduce natural difficulty in terms of common grounding while enabling easy evaluation and analysis of complex models. In this paper, we propose a minimal dialogue task which requires advanced skills of common grounding under continuous and partially-observable context. Based on this task formulation, we collected a largescale dataset of 6,760 dialogues which fulfills essential requirements of natural language corpora. Our analysis of the dataset revealed important phenomena related to common grounding that need to be considered. Finally, we evaluate and analyze baseline neural models on a simple subtask that requires recognition of the created common ground. We show that simple baseline models perform decently but leave room for further improvement. Overall, we show that our proposed task will be a fundamental testbed where we can train, evaluate, and analyze dialogue system's ability for sophisticated common grounding.

Computation and Language,Artificial Intelligence
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

Pradeep Dasigi,Nelson F. Liu,Ana Marasović,Noah A. Smith,Matt Gardner

DOI: https://doi.org/10.48550/arXiv.1908.05803

2019-09-05

Abstract:Machine comprehension of texts longer than a single sentence often requires coreference resolution. However, most current reading comprehension benchmarks do not contain complex coreferential phenomena and hence fail to evaluate the ability of models to resolve coreference. We present a new crowdsourced dataset containing more than 24K span-selection questions that require resolving coreference among entities in over 4.7K English paragraphs from Wikipedia. Obtaining questions focused on such phenomena is challenging, because it is hard to avoid lexical cues that shortcut complex reasoning. We deal with this issue by using a strong baseline model as an adversary in the crowdsourcing loop, which helps crowdworkers avoid writing questions with exploitable surface cues. We show that state-of-the-art reading comprehension models perform significantly worse than humans on this benchmark---the best model performance is 70.5 F1, while the estimated human performance is 93.4 F1.

Computation and Language
Evaluating Commonsense in Pre-trained Language Models

Xuhui Zhou,Yue Zhang,Leyang Cui,Dandan Huang

DOI: https://doi.org/10.48550/arXiv.1911.11931

2021-02-11

Abstract:Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.

Computation and Language,Artificial Intelligence
Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning

Xiangyu Peng,Siyan Li,Sarah Wiegreffe,Mark Riedl

2023-11-18

Abstract:Transformer-based language model approaches to automated story generation currently provide state-of-the-art results. However, they still suffer from plot incoherence when generating narratives over time, and critically lack basic commonsense reasoning. Furthermore, existing methods generally focus only on single-character stories, or fail to track characters at all. To improve the coherence of generated narratives and to expand the scope of character-centric narrative generation, we introduce Commonsense-inference Augmented neural StoryTelling (CAST), a framework for introducing commonsense reasoning into the generation process with the option to model the interaction between multiple characters. We find that our CAST method produces significantly more coherent, on-topic, enjoyable and fluent stories than existing models in both the single-character and two-character settings in three storytelling domains.

Computation and Language
Deriving Commonsense Inference Tasks from Interactive Fictions

Mo Yu,Xiaoxiao Guo,Yufei Feng,Xiaodan Zhu,Michael Greenspan,Murray Campbell

DOI: https://doi.org/10.48550/arXiv.2010.09788

2020-10-20

Abstract:Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an indispensable cornerstone in building general AI systems. We propose a new commonsense reasoning dataset based on human's interactive fiction game playings as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset mitigates several limitations of the prior art. Experiments show that our task is solvable to human experts with sufficient commonsense knowledge but poses challenges to existing machine reading models, with a big performance gap of more than 30%.

Artificial Intelligence,Computation and Language
Unsupervised Story Comprehension With Hierarchical Encoder-Decoder

Bingning Wang,Ting Yao,Qi Zhang,Jingfang Xu,Kang Liu,Zhixing Tian,Jun Zhao

DOI: https://doi.org/10.1145/3341981.3344227

2019-01-01

Abstract:Commonsense understanding is a long-term goal of natural language processing yet to be resolved. One standard testbed for commonsense understanding is Story Cloze Test (SCT) [22], In SCT, given a 4-sentences story, we are expected to select the proper ending out of two proposed candidates. The training set in SCT only contains unlabeled stories, previous works usually adopt the small labeled development data for training, which ignored the sufficient training data and, essentially, not reveal the commonsense reasoning procedure. In this paper, we propose an unsupervised sequence-to-sequence method for story reading comprehension, we only adopt the unlabeled story and directly model the context-target inference probability. We propose a loss-reweight training strategy for the seq-to-seq model to dynamically tuning the training process. Experimental results demonstrate the advantage of the proposed model and it achieves the comparable results with supervised methods on SCT.
Augmenting End-to-End Dialogue Systems with Commonsense Knowledge

Tom Young,Erik Cambria,Iti Chaturvedi,Hao Zhou,Subham Biswas,Minlie Huang

DOI: https://doi.org/10.1609/aaai.v32i1.11923

2018-01-01

Abstract:Building dialogue systems that can converse naturally with humans is a challenging yet intriguing problem of artificial intelligence. In open-domain human-computer conversation, where the conversational agent is expected to respond to human utterances in an interesting and engaging way, commonsense knowledge has to be integrated into the model effectively. In this paper, we investigate the impact of providing commonsense knowledge about the concepts covered in the dialogue. Our model represents the first attempt to integrating a large commonsense knowledge base into end-to end conversational models. In the retrieval-based scenario, we propose a model to jointly take into account message content and related commonsense for selecting an appropriate response. Our experiments suggest that the knowledge augmented models are superior to their knowledge-free counterparts.
What Really is Commonsense Knowledge?

Quyet V. Do,Junze Li,Tung-Duong Vuong,Zhaowei Wang,Yangqiu Song,Xiaojuan Ma

2024-11-06

Abstract:Commonsense datasets have been well developed in Natural Language Processing, mainly through crowdsource human annotation. However, there are debates on the genuineness of commonsense reasoning benchmarks. In specific, a significant portion of instances in some commonsense benchmarks do not concern commonsense knowledge. That problem would undermine the measurement of the true commonsense reasoning ability of evaluated models. It is also suggested that the problem originated from a blurry concept of commonsense knowledge, as distinguished from other types of knowledge. To demystify all of the above claims, in this study, we survey existing definitions of commonsense knowledge, ground into the three frameworks for defining concepts, and consolidate them into a multi-framework unified definition of commonsense knowledge (so-called consolidated definition). We then use the consolidated definition for annotations and experiments on the CommonsenseQA and CommonsenseQA 2.0 datasets to examine the above claims. Our study shows that there exists a large portion of non-commonsense-knowledge instances in the two datasets, and a large performance gap on these two subsets where Large Language Models (LLMs) perform worse on commonsense-knowledge instances.

Computation and Language,Artificial Intelligence
A Comprehensive Overview of CFN From a Commonsense Perspective

Ru Li,Yunxiao Zhao,Zhiqiang Wang,Xuefeng Su,Shaoru Guo,Yong Guan,Xiaoqi Han,Hongyan Zhao

DOI: https://doi.org/10.1007/s11633-023-1450-8

2024-01-01

Machine Intelligence Research

Abstract:Chinese FrameNet (CFN) is a scenario commonsense knowledge base (CKB) that plays an important role in research on Chinese language understanding. It is based on the theory of frame semantics and English FrameNet (FN). The CFN knowledge base contains a wealth of scenario commonsense knowledge, including frames, frame elements, and frame relations, as well as annotated instances with rich scenario-related labels on Chinese sentences and discourses. In this paper, we conduct a comprehensive overview of CFN from a commonsense perspective, covering topics such as scenario commonsense representation, CFN resources, and its applications. We also summarize recent breakthroughs and identify future research directions. First, we introduce the concept of scenario commonsense, including its definitions, examples, and representation methods, with a focus on the relationship between scenario commonsense and the frame concept in CFN. In addition, we provide a comprehensive overview of CFN resources and their applications, highlighting the newly proposed frame-based discourse representation and a human-machine collaboration framework for expanding the CFN corpus. Furthermore, we explore emerging topics such as expanding the CFN resource, improving the interpretability of machine reading comprehension, and using scenario CKBs for text generation.

A corpus and cloze evaluation for deeper understanding of commonsense stories

A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories

Building a commonsense knowledge base for context-awareness inference

Incorporating Structured Commonsense Knowledge in Story Completion

Paragraph-level Commonsense Transformers with Recurrent Memory

A Corpus for Understanding and Generating Moral Stories

Natural Language Processing with Commonsense Knowledge: A Survey

Event2Mind: Commonsense Inference on Events, Intents, and Reactions

CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models

A Manual Experiment On Commonsense Knowledge Acquisition From Web Corpora

TransOMCS: From Linguistic Graphs to Commonsense Knowledge

UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations

A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context

Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

Evaluating Commonsense in Pre-trained Language Models

Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning

Deriving Commonsense Inference Tasks from Interactive Fictions

Unsupervised Story Comprehension With Hierarchical Encoder-Decoder

Augmenting End-to-End Dialogue Systems with Commonsense Knowledge

What Really is Commonsense Knowledge?

A Comprehensive Overview of CFN From a Commonsense Perspective