Abstract:Topic segmentation is critical for obtaining structured documents and improving downstream tasks such as information retrieval. Due to its ability of automatically exploring clues of topic shift from abundant labeled data, recent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper relationship between coherence and topic segmentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives to further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). Specifically, the TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document, which is constructed by jointly disrupting the original document at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar. Extensive experiments show that the Longformer with our approach significantly outperforms old state-of-the-art (SOTA) methods. Our approach improve $F_1$ of old SOTA by 3.42 (73.74 -> 77.16) and reduces $P_k$ by 1.11 points (15.0 -> 13.89) on WIKI-727K and achieves an average relative reduction of 4.3% on $P_k$ on WikiSection. The average relative $P_k$ drop of 8.38% on two out-of-domain datasets also demonstrates the robustness of our approach.

Evaluating Text Coherence at Sentence and Paragraph Levels.

A Model of Coherence Based on Distributed Sentence Representation.

Text Coherence Analysis Based on Deep Neural Network.

A Novel Computational and Modeling Foundation for Automatic Coherence Assessment

A Hierarchical Neural Autoencoder for Paragraphs and Documents

Learning to Rank Semantic Coherence for Topic Segmentation.

Hierarchical Attention Networks for Sentence Ordering.

Learning to Order Natural Language Texts.

Analyzing Neural Discourse Coherence Models

Neural Sentence Ordering.

DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting

Neural Net Models for Open-Domain Discourse Coherence

Transformer Models for Text Coherence Assessment

Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence

Task-Related Pretraining with Whole Word Masking for Chinese Coherence Evaluation.

Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling

Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models

Improving the Generalization Ability in Essay Coherence Evaluation through Monotonic Constraints

Discourse Analysis for Evaluating Coherence in Video Paragraph Captions

A Study on the Distribution Differences of Sentence Group Coherence in Chinese Texts of Different Genres.

On Improving Text Generation Via Integrating Text Coherence