Abstract:Developing the utilized intelligent systems is increasingly important to learn effective text representations, especially extract the sentence features. Numerous previous studies have been concentrated on the task of sentence representation learning based on deep learning approaches. However, the present approaches are mostly proposed with the single task or replied on the labeled corpus when learning the embedding of the sentences. In this paper, we assess the factors in learning sentence representation and propose an efficient unsupervised learning framework with multi-task learning (USR-MTL), in which various text learning tasks are merged into the unitized framework. With the syntactic and semantic features of sentences, three different factors to some extent are reflected in the task of the sentence representation learning that is the wording, or the ordering of the neighbored sentences of a target sentence in other words. Hence, we integrate the word-order learning task, word prediction task, and the sentence-order learning task into the proposed framework to attain meaningful sentence embeddings. Here, the process of sentence embedding learning is reformulated as a multi-task learning framework of the sentence-level task and the two word-level tasks. Moreover, the proposed framework is motivated by an unsupervised learning algorithm utilizing the unlabeled corpus. Based on the experimental results, our approach achieves the state-of-the-art performances on the downstream natural language processing tasks compared to the popular unsupervised representation learning techniques. The experiments on representation visualization and task analysis demonstrate the effectiveness of the tasks in the proposed framework in creating reasonable sentence representations proving the capacity of the proposed unsupervised multi-task framework for the sentence representation learning.

Unsupervised Learning in a Framework of Information Compression by Multiple Alignment, Unification and Search

Information Compression by Multiple Alignment, Unification and Search as a Unifying Principle in Computing and Cognition

Probabilistic Reasoning as Information Compression by Multiple Alignment, Unification and Search: An Introduction and Overview

Medical diagnosis as pattern recognition in a framework of information compression by multiple alignment, unification and search

Unsupervised Face Alignment by Robust Nonrigid Mapping

Semantic Compression with Information Lattice Learning

Meta-Unsupervised-Learning: A supervised approach to unsupervised learning

Information Compression in the AI Era: Recent Advances and Future Challenges

Unsupervised Semantic Association Learning with Latent Label Inference.

Maintaining Structural Information by Pairwise Similarity for Unsupervised Domain Adaptation.

Image Coding for Machines with Omnipotent Feature Learning

Unsupervised Learning for 3D Ultrasonic Data Compression

Compressible and Searchable: AI-native Multi-Modal Retrieval System with Learned Image Compression

Structure Learning via Mutual Information

Unsupervised Language Acquisition

Computing as compression: the SP theory of intelligence

Unsupervised Domain Adaptation for Brain Structure Segmentation Via Mutual Information Maximization Alignment

Understanding is Compression

Unifying Demonstration Selection and Compression for In-Context Learning

A Multi-Modal Hashing Learning Framework for Automatic Image Annotation

Usr-mtl: an Unsupervised Sentence Representation Learning Framework with Multi-Task Learning