Abstract:The processing and comprehension of numerical information in natural language represent pivotal focal points of scholarly inquiry. Across diverse applications spanning text analysis to information retrieval, the adept management and understanding of the numerical content within natural language are indispensable in achieving task success. Specialized encoding and embedding techniques tailored to numerical data offer an avenue toward improved performance in tasks involving masked prediction and numerical reasoning, inherently characterized by numerical values. Consequently, treating numbers in text merely as words is inadequate; their numerical semantics must be underscored. Recent years have witnessed the emergence of a range of specific encoding methodologies designed explicitly for numerical content, demonstrating promising outcomes. We observe similarities between the Transformer architecture and CPU architecture, with symmetry playing a crucial role. In light of this observation and drawing inspiration from computer system theory, we introduce a floating-point representation and devise a corresponding embedding module. The numerical representations correspond one-to-one with their semantic vector values, rendering both symmetric regarding intermediate transformation methods. Our proposed methodology facilitates the more comprehensive encoding and embedding of numerical information within a predefined precision range, thereby ensuring a distinctive encoding representation for each numerical entity. Rigorous testing on multiple encoder-only models and datasets yielded results that stand out in terms of competitiveness. In comparison to the default embedding methods employed by models, our approach achieved an improvement of approximately 3.8% in Top-1 accuracy and a reduction in perplexity of approximately 0.43. These outcomes affirm the efficacy of our proposed method. Furthermore, the enrichment of numerical semantics through a more comprehensive embedding contributes to the augmentation of the model's capacity for semantic understanding.

Learning Numeral Embedding.

Learning Numeral Embeddings

Floating-Point Embedding: Enhancing the Mathematical Comprehension of Large Language Models

Interleaving Text and Number Embeddings to Solve Mathemathics Problems

Do NLP Models Know Numbers? Probing Numeracy in Embeddings.

Laying Anchors: Semantically Priming Numerals in Language Modeling

How to Leverage Digit Embeddings to Represent Numbers?

Estimating Numbers without Regression

Two-Stage Label Embedding Via Neural Factorization Machine for Multi-Label Classification

Numeral Understanding in Financial Tweets for Fine-Grained Crowd-Based Forecasting

Learning Effective Word Embedding Using Morphological Word Similarity

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

Pre-training and Evaluation of Numeracy-Oriented Language Model.

Delta Embedding Learning

NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining

Joint Learning of Character and Word Embeddings.

Learning Approximate and Exact Numeral Systems via Reinforcement Learning

EmbedLLM: Learning Compact Representations of Large Language Models

Nugget: Neural Agglomerative Embeddings of Text

Learning Mathematical Properties of Integers

From Categorical to Numerical: Multiple Transitive Distance Learning and Embedding.