Abstract:One fundamental task for NLP is to determine the similarity between two texts and evaluate the extent of their likeness. The previous methods for the Persian language have low accuracy and are unable to comprehend the structure and meaning of texts effectively. Additionally, these methods primarily focus on formal texts, but in real-world applications of text processing, there is a need for robust methods that can handle colloquial texts. This requires algorithms that consider the structure and significance of words based on context, rather than just the frequency of words. The lack of a proper dataset for this task in the Persian language makes it important to develop such algorithms and construct a dataset for Persian text. This paper introduces a new transformer-based model to measure semantic similarity between Persian informal short texts from social networks. In addition, a Persian dataset named FarSSiM has been constructed for this purpose, using real data from social networks and manually annotated and verified by a linguistic expert team. The proposed model involves training a large language model using the BERT architecture from scratch. This model, called FarSSiBERT, is pre-trained on approximately 104 million Persian informal short texts from social networks, making it one of a kind in the Persian language. Moreover, a novel specialized informal language tokenizer is provided that not only performs tokenization on formal texts well but also accurately identifies tokens that other Persian tokenizers are unable to recognize. It has been demonstrated that our proposed model outperforms ParsBERT, laBSE, and multilingual BERT in the Pearson and Spearman's coefficient criteria. Additionally, the pre-trained large language model has great potential for use in other NLP tasks on colloquial text and as a tokenizer for less-known informal words.

Persian Pronoun Resolution: Leveraging Neural Networks and Language Models

Review of coreference resolution in English and Persian

A Sequence-to-Sequence Approach for Arabic Pronoun Resolution

Persian Natural Language Inference: A Meta-learning approach

TookaBERT: A Step Forward for Persian NLU

Towards Transparency in Coreference Resolution: A Quantum-Inspired Approach

Multi-BERT: Leveraging Adapters and Prompt Tuning for Low-Resource Multi-Domain Adaptation

Persian Semantic Role Labeling Using Transfer Learning and BERT-Based Models

PERCORE: A Deep Learning-Based Framework for Persian Spelling Correction with Phonetic Analysis

A transformer-based deep learning model for Persian moral sentiment analysis

PEYMA: A Tagged Corpus for Persian Named Entities

FarSSiBERT: A Novel Transformer-based Model for Semantic Similarity Measurement of Persian Social Networks Informal Texts

The Recognition Of Persian Phonemes Using PPNet

FaBERT: Pre-training BERT on Persian Blogs

PersianRAG: A Retrieval-Augmented Generation System for Persian Language

A Hybrid Persian Sentiment Analysis Framework: Integrating Dependency Grammar Based Rules and Deep Neural Networks

A multimodal deep learning approach for named entity recognition from social media

Exploring the Potential of Machine Translation for Generating Named Entity Datasets: A Case Study between Persian and English

PersianLLaMA: Towards Building First Persian Large Language Model

Persian Slang Text Conversion to Formal and Deep Learning of Persian Short Texts on Social Media for Sentiment Classification