mlr3summary: Concise and interpretable summaries for machine learning models

Susanne Dandl,Marc Becker,Bernd Bischl,Giuseppe Casalicchio,Ludwig Bothmann

2024-04-25

Abstract:This work introduces a novel R package for concise, informative summaries of machine learning models.

Machine Learning

What problem does this paper attempt to address?

Model interpretation through lower-dimensional posterior summarization

Spencer Woody,Carlos M. Carvalho,Jared S. Murray

DOI: https://doi.org/10.1080/10618600.2020.1796684

2020-02-29

Abstract:Nonparametric regression models have recently surged in their power and popularity, accompanying the trend of increasing dataset size and complexity. While these models have proven their predictive ability in empirical settings, they are often difficult to interpret and do not address the underlying inferential goals of the analyst or decision maker. In this paper, we propose a modular two-stage approach for creating parsimonious, interpretable summaries of complex models which allow freedom in the choice of modeling technique and the inferential target. In the first stage a flexible model is fit which is believed to be as accurate as possible. In the second stage, lower-dimensional summaries are constructed by projecting draws from the distribution onto simpler structures. These summaries naturally come with valid Bayesian uncertainty estimates. Further, since we use the data only once to move from prior to posterior, these uncertainty estimates remain valid across multiple summaries and after iteratively refining a summary. We apply our method and demonstrate its strengths across a range of simulated and real datasets. Code to reproduce the examples shown is avaiable at <a class="link-external link-http" href="http://github.com/spencerwoody/ghost" rel="external noopener nofollow">this http URL</a>

Methodology
mlr3proba: an R package for machine learning in survival analysis

Raphael Sonabend,Franz J Király,Andreas Bender,Bernd Bischl,Michel Lang,Franz J. Király

DOI: https://doi.org/10.1093/bioinformatics/btab039

IF: 5.8

2021-02-01

Bioinformatics

Abstract:ABSTRACT Summary As machine learning has become increasingly popular over the last few decades, so too has the number of machine-learning interfaces for implementing these models. Whilst many R libraries exist for machine learning, very few offer extended support for survival analysis. This is problematic considering its importance in fields like medicine, bioinformatics, economics, engineering and more. mlr3proba provides a comprehensive machine-learning interface for survival analysis and connects with mlr3’s general model tuning and benchmarking facilities to provide a systematic infrastructure for survival modelling and evaluation. Availability and implementation mlr3proba is available under an LGPL-3 licence on CRAN and at https://github.com/mlr-org/mlr3proba, with further documentation at https://mlr3book.mlr-org.com/survival.html.

biochemical research methods,biotechnology & applied microbiology,mathematical & computational biology
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Jesse Vig,Wojciech Kryściński,Karan Goel,Nazneen Fatema Rajani

DOI: https://doi.org/10.48550/arXiv.2104.07605

2021-07-26

Abstract:Novel neural architectures, training strategies, and the availability of large-scale corpora haven been the driving force behind recent progress in abstractive text summarization. However, due to the black-box nature of neural models, uninformative evaluation metrics, and scarce tooling for model and data analysis, the true performance and failure modes of summarization models remain largely unknown. To address this limitation, we introduce SummVis, an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of the models, data, and evaluation metrics associated with text summarization. Through its lexical and semantic visualizations, the tools offers an easy entry point for in-depth model prediction exploration across important dimensions such as factual consistency or abstractiveness. The tool together with several pre-computed model outputs is available at <a class="link-external link-https" href="https://github.com/robustness-gym/summvis" rel="external noopener nofollow">this https URL</a>.

Computation and Language
Improving Expert Radiology Report Summarization by Prompting Large Language Models with a Layperson Summary

Xingmeng Zhao,Tongnian Wang,Anthony Rios

2024-06-21

Abstract:Radiology report summarization (RRS) is crucial for patient care, requiring concise "Impressions" from detailed "Findings." This paper introduces a novel prompting strategy to enhance RRS by first generating a layperson summary. This approach normalizes key observations and simplifies complex information using non-expert communication techniques inspired by doctor-patient interactions. Combined with few-shot in-context learning, this method improves the model's ability to link general terms to specific findings. We evaluate this approach on the MIMIC-CXR, CheXpert, and MIMIC-III datasets, benchmarking it against 7B/8B parameter state-of-the-art open-source large language models (LLMs) like Meta-Llama-3-8B-Instruct. Our results demonstrate improvements in summarization accuracy and accessibility, particularly in out-of-domain tests, with improvements as high as 5% for some metrics.

Computation and Language
TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale

Pengcheng Jiang,Cao Xiao,Zifeng Wang,Parminder Bhatia,Jimeng Sun,Jiawei Han

2024-03-15

Abstract:The advent of large language models (LLMs) has significantly advanced natural language processing tasks like text summarization. However, their large size and computational demands, coupled with privacy concerns in data transmission, limit their use in resource-constrained and privacy-centric settings. To overcome this, we introduce TriSum, a framework for distilling LLMs' text summarization abilities into a compact, local model. Initially, LLMs extract a set of aspect-triple rationales and summaries, which are refined using a dual-scoring method for quality. Next, a smaller local model is trained with these tasks, employing a curriculum learning strategy that evolves from simple to complex tasks. Our method enhances local model performance on various benchmarks (CNN/DailyMail, XSum, and ClinicalTrial), outperforming baselines by 4.5%, 8.5%, and 7.4%, respectively. It also improves interpretability by providing insights into the summarization rationale.

Computation and Language
RTSUM: Relation Triple-based Interpretable Summarization with Multi-level Salience Visualization

Seonglae Cho,Yonggi Cho,HoonJae Lee,Myungha Jang,Jinyoung Yeo,Dongha Lee

2024-03-25

Abstract:In this paper, we present RTSUM, an unsupervised summarization framework that utilizes relation triples as the basic unit for summarization. Given an input document, RTSUM first selects salient relation triples via multi-level salience scoring and then generates a concise summary from the selected relation triples by using a text-to-text language model. On the basis of RTSUM, we also develop a web demo for an interpretable summarizing tool, providing fine-grained interpretations with the output summary. With support for customization options, our tool visualizes the salience for textual units at three distinct levels: sentences, relation triples, and phrases. The codes,are publicly available.

Machine Learning
Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees

Swarnadeep Saha,Shiyue Zhang,Peter Hase,Mohit Bansal

DOI: https://doi.org/10.48550/arXiv.2209.10492

2022-09-21

Computation and Language

Abstract:Current abstractive summarization models either suffer from a lack of clear interpretability or provide incomplete rationales by only highlighting parts of the source document. To this end, we propose the Summarization Program (SP), an interpretable modular framework consisting of an (ordered) list of binary trees, each encoding the step-by-step generative process of an abstractive summary sentence from the source document. A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence (root node) to the document sentences (leaf nodes) from which it is derived, with the connecting nodes containing intermediate generated sentences. Edges represent different modular operations involved in summarization such as sentence fusion, compression, and paraphrasing. We first propose an efficient best-first search method over neural modules, SP-Search that identifies SPs for human summaries by directly optimizing for ROUGE scores. Next, using these programs as automatic supervision, we propose seq2seq models that generate Summarization Programs, which are then executed to obtain final summaries. We demonstrate that SP-Search effectively represents the generative process behind human summaries using modules that are typically faithful to their intended behavior. We also conduct a simulation study to show that Summarization Programs improve the interpretability of summarization models by allowing humans to better simulate model reasoning. Summarization Programs constitute a promising step toward interpretable and modular abstractive summarization, a complex task previously addressed primarily through blackbox end-to-end neural systems. Supporting code available at https://github.com/swarnaHub/SummarizationPrograms
RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Texts

Yuelyu Ji,Zhuochun Li,Rui Meng,Sonish Sivarajkumar,Yanshan Wang,Zeshui Yu,Hui Ji,Yushui Han,Hanyu Zeng,Daqing He

2024-06-25

Abstract:This paper introduces the RAG-RLRC-LaySum framework, designed to make complex biomedical research understandable to laymen through advanced Natural Language Processing (NLP) techniques. Our Retrieval Augmented Generation (RAG) solution, enhanced by a reranking method, utilizes multiple knowledge sources to ensure the precision and pertinence of lay summaries. Additionally, our Reinforcement Learning for Readability Control (RLRC) strategy improves readability, making scientific content comprehensible to non-specialists. Evaluations using the publicly accessible PLOS and eLife datasets show that our methods surpass Plain Gemini model, demonstrating a 20% increase in readability scores, a 15% improvement in ROUGE-2 relevance scores, and a 10% enhancement in factual accuracy. The RAG-RLRC-LaySum framework effectively democratizes scientific knowledge, enhancing public engagement with biomedical discoveries.

Computation and Language
Globally-Consistent Rule-Based Summary-Explanations for Machine Learning Models: Application to Credit-Risk Evaluation

Cynthia Rudin,Yaron Shaposhnik

DOI: https://doi.org/10.2139/ssrn.3395422

2019-01-01

SSRN Electronic Journal

Abstract:We develop a method for interpreting specific predictions made by (global) predictive models by constructing (local) models tailored to each specific observation (these are also called "explanations" in the literature). Unlike existing work that "explains'' specific observations by approximating global models in the vicinity of these observations, we fit models that are globally consistent with predictions made by the global model on past data. We focus on rule-based models (also known as association rules or conjunctions of predicates), which are interpretable and widely used in practice. We design multiple algorithms to extract such rules from discrete and continuous datasets, and study their theoretical properties. Finally, we apply these algorithms to multiple credit-risk models trained on real-world data from FICO and demonstrate that our approach effectively produces sparse summary-explanations of these models in short period of time. Our approach is model-agnostic (that is, can be used to interpret any predictive model), and solves a minimum set cover problem to construct its summaries.
fairml: A Statistician's Take on Fair Machine Learning Modelling

Marco Scutari

2023-05-03

Abstract:The adoption of machine learning in applications where it is crucial to ensure fairness and accountability has led to a large number of model proposals in the literature, largely formulated as optimisation problems with constraints reducing or eliminating the effect of sensitive attributes on the response. While this approach is very flexible from a theoretical perspective, the resulting models are somewhat black-box in nature: very little can be said about their statistical properties, what are the best practices in their applied use, and how they can be extended to problems other than those they were originally designed for. Furthermore, the estimation of each model requires a bespoke implementation involving an appropriate solver which is less than desirable from a software engineering perspective. In this paper, we describe the fairml R package which implements our previous work (Scutari, Panero, and Proissl 2022) and related models in the literature. fairml is designed around classical statistical models (generalised linear models) and penalised regression results (ridge regression) to produce fair models that are interpretable and whose properties are well-known. The constraint used to enforce fairness is orthogonal to model estimation, making it possible to mix-and-match the desired model family and fairness definition for each application. Furthermore, fairml provides facilities for model estimation, model selection and validation including diagnostic plots.

Machine Learning,Computers and Society
Learning to Summarize from LLM-generated Feedback

Hwanjun Song,Taewon Yun,Yuho Lee,Gihun Lee,Jason Cai,Hang Su

2024-10-17

Abstract:Developing effective text summarizers remains a challenge due to issues like hallucinations, key information omissions, and verbosity in LLM-generated summaries. This work explores using LLM-generated feedback to improve summary quality by aligning the summaries with human preferences for faithfulness, completeness, and conciseness. We introduce FeedSum, a large-scale dataset containing multi-dimensional LLM feedback on summaries of varying quality across diverse domains. Our experiments show how feedback quality, dimensionality, and granularity influence preference learning, revealing that high-quality, multi-dimensional, fine-grained feedback significantly improves summary generation. We also compare two methods for using this feedback: supervised fine-tuning and direct preference optimization. Finally, we introduce SummLlama3-8b, a model that outperforms the nearly 10x larger Llama3-70b-instruct in generating human-preferred summaries, demonstrating that smaller models can achieve superior performance with appropriate training. The full dataset will be released soon. The SummLlama3-8B model is now available at <a class="link-external link-https" href="https://huggingface.co/DISLab/SummLlama3-8B" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence
Information-Theoretic Distillation for Reference-less Summarization

Jaehun Jung,Ximing Lu,Liwei Jiang,Faeze Brahman,Peter West,Pang Wei Koh,Yejin Choi

2024-08-20

Abstract:The current winning recipe for automatic summarization is using proprietary large-scale language models (LLMs) such as ChatGPT as is, or imitation learning from them as teacher models. While increasingly ubiquitous dependence on such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek an alternative learning method -- that allows for a more cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization, without relying on either the LLM's capability or human-written references. To achieve this, we first propose a novel formulation of the desiderata of summarization (saliency, faithfulness and brevity) through the lens of mutual information between the original document and the summary. Based on this formulation, we start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization, then self-train the model to optimize for the information-centric measures of ideal summaries. Distilling from the improved teacher, we arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive analysis demonstrates that our approach outperforms in-domain supervised models in human evaluation, let alone state-of-the-art unsupervised methods, and wins over ChatGPT in controllable summarization.

Computation and Language,Artificial Intelligence
RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs

Jiaxing Wu,Lin Ning,Luyang Liu,Harrison Lee,Neo Wu,Chao Wang,Sushant Prakash,Shawn O'Banion,Bradley Green,Jun Xie

2024-09-07

Abstract:LLM-powered personalization agent systems employ Large Language Models (LLMs) to predict users' behavior from their past activities. However, their effectiveness often hinges on the ability to effectively leverage extensive, long user historical data due to its inherent noise and length of such data. Existing pretrained LLMs may generate summaries that are concise but lack the necessary context for downstream tasks, hindering their utility in personalization systems. To address these challenges, we introduce Reinforcement Learning from Prediction Feedback (RLPF). RLPF fine-tunes LLMs to generate concise, human-readable user summaries that are optimized for downstream task performance. By maximizing the usefulness of the generated summaries, RLPF effectively distills extensive user history data while preserving essential information for downstream tasks. Our empirical evaluation demonstrates significant improvements in both extrinsic downstream task utility and intrinsic summary quality, surpassing baseline methods by up to 22% on downstream task performance and achieving an up to 84.59% win rate on Factuality, Abstractiveness, and Readability. RLPF also achieves a remarkable 74% reduction in context length while improving performance on 16 out of 19 unseen tasks and/or datasets, showcasing its generalizability. This approach offers a promising solution for enhancing LLM personalization by effectively transforming long, noisy user histories into informative and human-readable representations.

Computation and Language,Artificial Intelligence,Machine Learning
Enhancing Trust in LLM-Generated Code Summaries with Calibrated Confidence Scores

Yuvraj Virk,Premkumar Devanbu,Toufique Ahmed

2024-04-30

Abstract:A good summary can often be very useful during program comprehension. While a brief, fluent, and relevant summary can be helpful, it does require significant human effort to produce. Often, good summaries are unavailable in software projects, thus making maintenance more difficult. There has been a considerable body of research into automated AI-based methods, using Large Language models (LLMs), to generate summaries of code; there also has been quite a bit work on ways to measure the performance of such summarization methods, with special attention paid to how closely these AI-generated summaries resemble a summary a human might have produced. Measures such as BERTScore and BLEU have been suggested and evaluated with human-subject studies.

Software Engineering,Computation and Language
Explaining black box text modules in natural language with language models

Chandan Singh,Aliyah R. Hsu,Richard Antonello,Shailee Jain,Alexander G. Huth,Bin Yu,Jianfeng Gao

2023-11-16

Abstract:Large language models (LLMs) have demonstrated remarkable prediction performance for a growing array of tasks. However, their rapid proliferation and increasing opaqueness have created a growing need for interpretability. Here, we ask whether we can automatically obtain natural language explanations for black box text modules. A "text module" is any function that maps text to a scalar continuous value, such as a submodule within an LLM or a fitted model of a brain region. "Black box" indicates that we only have access to the module's inputs/outputs. We introduce Summarize and Score (SASC), a method that takes in a text module and returns a natural language explanation of the module's selectivity along with a score for how reliable the explanation is. We study SASC in 3 contexts. First, we evaluate SASC on synthetic modules and find that it often recovers ground truth explanations. Second, we use SASC to explain modules found within a pre-trained BERT model, enabling inspection of the model's internals. Finally, we show that SASC can generate explanations for the response of individual fMRI voxels to language stimuli, with potential applications to fine-grained brain mapping. All code for using SASC and reproducing results is made available on Github.

Artificial Intelligence,Computation and Language,Machine Learning,Neurons and Cognition
Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

Blair Yang,Fuyang Cui,Keiran Paster,Jimmy Ba,Pashootan Vaezipoor,Silviu Pitis,Michael R. Zhang

2024-09-02

Abstract:The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.

Machine Learning,Artificial Intelligence,Computation and Language
On the State of German (Abstractive) Text Summarization

Dennis Aumiller,Jing Fan,Michael Gertz

DOI: https://doi.org/10.48550/arXiv.2301.07095

2023-01-18

Abstract:With recent advancements in the area of Natural Language Processing, the focus is slowly shifting from a purely English-centric view towards more language-specific solutions, including German. Especially practical for businesses to analyze their growing amount of textual data are text summarization systems, which transform long input documents into compressed and more digestible summary texts. In this work, we assess the particular landscape of German abstractive text summarization and investigate the reasons why practically useful solutions for abstractive text summarization are still absent in industry. Our focus is two-fold, analyzing a) training resources, and b) publicly available summarization systems. We are able to show that popular existing datasets exhibit crucial flaws in their assumptions about the original sources, which frequently leads to detrimental effects on system generalization and evaluation biases. We confirm that for the most popular training dataset, MLSUM, over 50% of the training set is unsuitable for abstractive summarization purposes. Furthermore, available systems frequently fail to compare to simple baselines, and ignore more effective and efficient extractive summarization approaches. We attribute poor evaluation quality to a variety of different factors, which are investigated in more detail in this work: A lack of qualitative (and diverse) gold data considered for training, understudied (and untreated) positional biases in some of the existing datasets, and the lack of easily accessible and streamlined pre-processing strategies or analysis tools. We provide a comprehensive assessment of available models on the cleaned datasets, and find that this can lead to a reduction of more than 20 ROUGE-1 points during evaluation. The code for dataset filtering and reproducing results can be found online at <a class="link-external link-https" href="https://github.com/dennlinger/summaries" rel="external noopener nofollow">this https URL</a>

Computation and Language
Interpretable extractive text summarization with meta-learning and BI-LSTM: A study of meta learning and explainability techniques

Song-Nguyen Vo,Tien-Thinh Vo,Bac Le

DOI: https://doi.org/10.1016/j.eswa.2023.123045

IF: 8.5

2024-01-08

Expert Systems with Applications

Abstract:Text summarization is a widely-researched problem among scholars in the field of natural language processing. Multiple techniques have been proposed to help tackle this problem, yet some of these methodologies may still exhibit limitations such as the requirements for large training datasets, which might not always be possible, but more importantly, the lack of interpretability or transparency of the model. In this paper, we propose using meta-learning algorithm to train a deep learning model for extractive text summarization and then using various explanatory techniques such as SHAP (Shapley, 1953), linear regression (Lederer, 2022), decision trees (Fürnkranz, 2010), and input modification to gain insights into the model's decision making process. To evaluate the effectiveness of our approach, we will compare it to other popular natural language processing models like BERT (Miller, 2019) or XLNET (Yang et al., 2020) using the ROUGE metrics (Lin, 2004). Overall, our proposed approach provides a promising solution to the limitations of existing methods and a framework for improving the explainability of deep learning models in other natural language processing tasks.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
SUBPLEX: Towards a Better Understanding of Black Box Model Explanations at the Subpopulation Level

Jun Yuan,Gromit Yeuk-Yin Chan,Brian Barr,Kyle Overton,Kim Rees,Luis Gustavo Nonato,Enrico Bertini,Claudio T. Silva

2024-05-05

Abstract:Understanding the interpretation of machine learning (ML) models has been of paramount importance when making decisions with societal impacts such as transport control, financial activities, and medical diagnosis. While current model interpretation methodologies focus on using locally linear functions to approximate the models or creating self-explanatory models that give explanations to each input instance, they do not focus on model interpretation at the subpopulation level, which is the understanding of model interpretations across different subset aggregations in a dataset. To address the challenges of providing explanations of an ML model across the whole dataset, we propose SUBPLEX, a visual analytics system to help users understand black-box model explanations with subpopulation visual analysis. SUBPLEX is designed through an iterative design process with machine learning researchers to address three usage scenarios of real-life machine learning tasks: model debugging, feature selection, and bias detection. The system applies novel subpopulation analysis on ML model explanations and interactive visualization to explore the explanations on a dataset with different levels of granularity. Based on the system, we conduct user evaluation to assess how understanding the interpretation at a subpopulation level influences the sense-making process of interpreting ML models from a user's perspective. Our results suggest that by providing model explanations for different groups of data, SUBPLEX encourages users to generate more ingenious ideas to enrich the interpretations. It also helps users to acquire a tight integration between programming workflow and visual analytics workflow. Last but not least, we summarize the considerations observed in applying visualization to machine learning interpretations.

Human-Computer Interaction
Demystifying Code Summarization Models.

Yu Wang,Fengjuan Gao,Linzhang Wang

2021-01-01

Abstract:The last decade has witnessed a rapid advance in machine learningmodels.While the black-box nature of these systems allows powerful predictions, it cannot be directly explained, posing a threat to the continuing democratization of machine learning technology. Tackling the challenge of model explainability, research has made significant progress in demystifying the image classification models. In the same spirit of these works, this paper studies code summarizationmodels, particularly, given an input program for which a model makes a prediction, our goal is to reveal the key features that the model uses for predicting the label of the program. We realize our approach in HouYi, which we use to evaluate four prominent code summarization models: extreme summarizer, code2vec, code2seq, and sequence GNN. Results show that all models base their predictions on syntactic and lexical properties with little to none semantic implication. Based on this finding, we present a novel approach to explaining the predictions of code summarization models through the lens of training data. Ourwork opens up this exciting, new direction of studying what models have learned from source code.

mlr3summary: Concise and interpretable summaries for machine learning models

Model interpretation through lower-dimensional posterior summarization

mlr3proba: an R package for machine learning in survival analysis

SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Improving Expert Radiology Report Summarization by Prompting Large Language Models with a Layperson Summary

TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale

RTSUM: Relation Triple-based Interpretable Summarization with Multi-level Salience Visualization

Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees

RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Texts

Globally-Consistent Rule-Based Summary-Explanations for Machine Learning Models: Application to Credit-Risk Evaluation

fairml: A Statistician's Take on Fair Machine Learning Modelling

Learning to Summarize from LLM-generated Feedback

Information-Theoretic Distillation for Reference-less Summarization

RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs

Enhancing Trust in LLM-Generated Code Summaries with Calibrated Confidence Scores

Explaining black box text modules in natural language with language models

Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

On the State of German (Abstractive) Text Summarization

Interpretable extractive text summarization with meta-learning and BI-LSTM: A study of meta learning and explainability techniques

SUBPLEX: Towards a Better Understanding of Black Box Model Explanations at the Subpopulation Level

Demystifying Code Summarization Models.