Abstract:A distinction is often drawn between a model's ability to predict a label for an evaluation sample that is directly memorised from highly similar training samples versus an ability to predict the label via some method of generalisation. In the context of using Language Models for question-answering, discussion continues to occur as to the extent to which questions are answered through memorisation. We consider this issue for questions that would ideally be answered through reasoning over an associated context. We propose a method of identifying evaluation samples for which it is very unlikely our model would have memorised the answers. Our method is based on semantic similarity of input tokens and label tokens between training and evaluation samples. We show that our method offers advantages upon some prior approaches in that it is able to surface evaluation-train pairs that have overlap in either contiguous or discontiguous sequences of tokens. We use this method to identify unmemorisable subsets of our evaluation datasets. We train two Language Models in a multitask fashion whereby the second model differs from the first only in that it has two additional datasets added to the training regime that are designed to impart simple numerical reasoning strategies of a sort known to improve performance on some of our evaluation datasets but not on others. We then show that there is performance improvement between the two models on the unmemorisable subsets of the evaluation datasets that were expected to benefit from the additional training datasets. Specifically, performance on unmemorisable subsets of two of our evaluation datasets, DROP and ROPES significantly improves by 9.0%, and 25.7% respectively while other evaluation datasets have no significant change in performance.

Unsupervised Natural Question Answering with a Small Model

Discovering Latent Knowledge in Language Models Without Supervision

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions

Leveraging Large Language Models for Multiple Choice Question Answering

Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds

Do Smaller Language Models Answer Contextualised Questions Through Memorisation Or Generalisation?

Elaboration-Generating Commonsense Question Answering at Scale

Training Question Answering Models From Synthetic Data

Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs

Language Models are Unsupervised Multitask Learners

Efficient Medical Question Answering with Knowledge-Augmented Question Generation

Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrieval

Self-QA: Unsupervised Knowledge Guided Language Model Alignment

Enhancing Large Language Models with Knowledge Graphs for Robust Question Answering

Do Large Language Models Know What They Don't Know?

Large Language Models Meet Knowledge Graphs to Answer Factoid Questions

Syntax-informed Question Answering with Heterogeneous Graph Transformer

Long-Term Memory Networks for Question Answering

Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts

DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries

Question-Answering of UGC