Abstract:In scientific literature and industry, semantic and context-aware Natural Language Processing-based solutions have been gaining importance in recent years. The possibilities and performance shown by these models when dealing with complex Human Language Understanding tasks are unquestionable, from conversational agents to the fight against disinformation in social networks. In addition, considerable attention is also being paid to developing multilingual models to tackle the language bottleneck. An increase in size has accompanied the growing need to provide more complex models implementing all these features without being conservative in the number of dimensions required. This paper aims to provide a comprehensive account of the impact of a wide variety of dimensional reduction techniques on the performance of different state-of-the-art multilingual siamese transformers, including unsupervised dimensional reduction techniques such as linear and nonlinear feature extraction, feature selection, and manifold techniques. In order to evaluate the effects of these techniques, we considered the multilingual extended version of Semantic Textual Similarity Benchmark (mSTSb) and two different baseline approaches, one using the embeddings from the pre-trained version of five models and another using their fine-tuned STS version. The results evidence that it is possible to achieve an average reduction of 91.58 % ± 2.59 % in the number of dimensions of embeddings from pre-trained models requiring a fitting time 96.68 % ± 0.68 % faster than the fine-tuning process. Besides, we achieve 54.65 % ± 32.20 % dimensionality reduction in embeddings from fine-tuned models. The results of this study will significantly contribute to the understanding of how different tuning approaches affect performance on semantic-aware tasks and how dimensional reduction techniques deal with the high-dimensional embeddings computed for the STS task and their potential for other highly demanding NLP tasks.

Transformer-based dimensionality reduction

DMT-EV: an Explainable Deep Network for Dimension Reduction.

Deep Manifold Transformation for Dimension Reduction

UDRN: Unified Dimensional Reduction Neural Network for Feature Selection and Feature Projection

Invertible Manifold Learning for Dimension Reduction

An Intrinsic Dimension Perspective of Transformers for Sequential Modeling

DeepViT: Towards Deeper Vision Transformer

Vision Transformer: Vit and its Derivatives

A Survey on Vision Transformer

A Survey on Visual Transformer

DRCT: Saving Image Super-resolution away from Information Bottleneck

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

A Survey of Visual Transformers

VDTR: Video Deblurring with Transformer

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training.

DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract)

DifFormer: Multi-Resolutional Differencing Transformer With Dynamic Ranging for Time Series Analysis

Exploring Dimensionality Reduction Techniques in Multilingual Transformers

DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

Mixture of Hidden-Dimensions Transformer