Abstract:Emojis have become an integral part of digital communication, enriching text by conveying emotions, tone, and intent. Existing emoji recommendation methods are primarily evaluated based on their ability to match the exact emoji a user chooses in the original text. However, they ignore the essence of users' behavior on social media in that each text can correspond to multiple reasonable emojis. To better assess a model's ability to align with such real-world emoji usage, we propose a new semantics preserving evaluation framework for emoji recommendation, which measures a model's ability to recommend emojis that maintain the semantic consistency with the user's text. To evaluate how well a model preserves semantics, we assess whether the predicted affective state, demographic profile, and attitudinal stance of the user remain unchanged. If these attributes are preserved, we consider the recommended emojis to have maintained the original semantics. The advanced abilities of Large Language Models (LLMs) in understanding and generating nuanced, contextually relevant output make them well-suited for handling the complexities of semantics preserving emoji recommendation. To this end, we construct a comprehensive benchmark to systematically assess the performance of six proprietary and open-source LLMs using different prompting techniques on our task. Our experiments demonstrate that GPT-4o outperforms other LLMs, achieving a semantics preservation score of 79.23%. Additionally, we conduct case studies to analyze model biases in downstream classification tasks and evaluate the diversity of the recommended emojis.

Capturing Semantics for Imputation with Pre-trained Language Models

Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training

Explaining Language Models' Predictions with High-Impact Concepts

Probing Linguistic Information For Logical Inference In Pre-trained Language Models

Introducing Semantics into Speech Encoders.

Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

Towards Understanding What Code Language Models Learned

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

Semantics-Aware Inferential Network for Natural Language Understanding

Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics

Semantics Preserving Emoji Recommendation with Large Language Models

Sem4SAP: Synonymous Expression Mining From Open Knowledge Graph For Language Model Synonym-Aware Pretraining

Modeling Semantic Compositionality With Sememe Knowledge

Infusing Prompts with Syntax and Semantics

Analyzing the Role of Semantic Representations in the Era of Large Language Models

Semantic Importance-Aware Communications with Semantic Correction Using Large Language Models

Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders

Interpreting vision and language generative models with semantic visual priors

FiLM: Fill-in Language Models for Any-Order Generation

Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

Enabling Language Models to Fill in the Blanks