Abstract:Rigorous and reproducible evaluation is critical for assessing the state of the art and for guiding scientific advances in Artificial Intelligence. Evaluation is challenging in practice due to several reasons, including benchmark saturation, lack of transparency in methods used for measurement, development challenges in extracting measurements for generative tasks, and, more generally, the extensive number of capabilities required for a well-rounded comparison across models. We make three contributions to alleviate the above challenges. First, we present Eureka, an open-source framework for standardizing evaluations of large foundation models beyond single-score reporting and rankings. Second, we introduce Eureka-Bench as an extensible collection of benchmarks testing capabilities that (i) are still challenging for state-of-the-art models and (ii) represent fundamental but overlooked language and multimodal capabilities. The inherent space for improvement in non-saturated benchmarks enables us to discover meaningful differences between models at a capability level. Third, using Eureka, we conduct an analysis of 12 state-of-the-art models, providing in-depth insights into failure understanding and model comparison, which can be leveraged to plan targeted improvements. In contrast to recent trends in reports and leaderboards showing absolute rankings and claims for one model or another to be the best, our analysis shows that there is no such best model. Different models have different strengths, but there are models that appear more often than others as best performers for some capabilities. Despite the recent improvements, current models still struggle with several fundamental capabilities including detailed image understanding, benefiting from multimodal input when available rather than fully relying on language, factuality and grounding for information retrieval, and over refusals.

EUREKA: EUphemism Recognition Enhanced through Knn-based methods and Augmentation

TEDB System Description to a Shared Task on Euphemism Detection 2022

Impromptu Cybercrime Euphemism Detection

Exploring Euphemism Detection in Few-Shot and Zero-Shot Settings

A Report on the Euphemisms Detection Shared Task

Euphemistic Phrase Detection by Masked Language Model

MEDs for PETs: Multilingual Euphemism Disambiguation for Potentially Euphemistic Terms

kpfriends at SemEval-2022 Task 2: NEAMER -- Named Entity Augmented Multi-word Expression Recognizer

Turkish Delights: a Dataset on Turkish Euphemisms

FEED PETs: Further Experimentation and Expansion on the Disambiguation of Potentially Euphemistic Terms

Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech

Iterative Strategy for Named Entity Recognition with Imperfect Annotations.

Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media

CATs are Fuzzy PETs: A Corpus and Analysis of Potentially Euphemistic Terms

Corpus Analysis And Automatic Detection Of Emotion-Inducing Keywords

E-bidNER: A Two-stage Enhanced Named Entity Recognition for Chinese Bid Announcements

Eureka: Evaluating and Understanding Large Foundation Models

Data Augmentation for Hypernymy Detection

E3 : Keyphrase based News Event Exploration Engine

DEXTER: Deep Encoding of External Knowledge for Named Entity Recognition in Virtual Assistants

Named Entity Recognition with Bidirectional LSTM-CNNs