Abstract:Driven by the surge in students sharing insights on online platforms, evaluating discussion forum posts has emerged as a significant research focus. This assessment is crucial for both students and educators, as it enables teachers to efficiently gauge student comprehension and helps students tailor their learning approaches to suit their individual needs, thereby fostering personalized growth. However, existing research on post-quality assessment faces challenges: 1) They overlook multimodal information interaction between post and topic, leading to inaccurate evaluations. 2) They are usually classification tasks that fail to provide feedback on the minor and relative differences between posts on the same topic. 3) They are usually evaluated using a single perspective, which is insufficient for capturing the complexity of the relationship between post and topic. Based on the above challenges, we propose a new assessment task, Multimodal Topic-Post Relevance Score Prediction (MTRSP), which analyzes whether a student's post comprehensively answers the question of the discussion topic by combining text and images to predict topic-post relevance scores, i.e., the degree of correctness of the answer. We develop an end-to-end Multi-perspective Topic-Post Relevance Score Reasoning (MTRSR) Method to solve the MTRSP Task, which leverages images and text from both the post and topic to infer topic-post relevance scores based on semantic similarity and logical coherence. Specifically, the topic-post content relevance reasoning module uses multimodal fusion to learn the semantic similarity of posts and topics. The logical coherence inference module examines the logical connections between posts and topics. Finally, we use three newly collected multimodal topic-post datasets and the public dataset Lazada-Home as an evaluation benchmark for the MTRSP task. Experimental results show that our MTRSR method can bring up to 9.02% in the NDCG@3 (Normalized Discounted Cumulative Gain) metric compared to the best-performing text-only model. The source code and dataset will be made public.

Exploring Relevance and Coherence for Automated Text Scoring Using Multi-task Learning

Coh-Metrix Model-Based Automatic Assessment of Interpreting Quality

End-to-end Multi-perspective Multimodal Posts Relevance Score Reasoning Prediction

A Novel Computational and Modeling Foundation for Automatic Coherence Assessment

Semantic Coherence Analysis of English Texts Based on Sentence Semantic Graphs

A Multi-task Automated Assessment System for Essay Scoring

Task-Related Pretraining with Whole Word Masking for Chinese Coherence Evaluation.

Enhancing Coherence of Extractive Summarization with Multitask Learning

Cross-Topic Text Auto-Scoring Method Based on Generative Adversarial Networks

Automated Quality Assessment of Web Pages from Textual Content

Human-machine Collaborative Scoring of Subjective Assignments Based on Sequential Three-Way Decisions

Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding.

CoAScore: Chain-of-Aspects Prompting for NLG Evaluation

BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence

Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances

BatchEval: Towards Human-like Text Evaluation

A study of linguistic characteristics and score predictors for different reading-to-write tasks

Student Programs Performance Scoring Based on Probabilistic Latent Semantic Analysis and Multi-granularity Feature Fusion for MOOC

Chinese Argumentative Writing Quality Evaluation Based on Multi-perspective Modeling

Automated Speaking Assessment of Conversation Tests with Novel Graph-based Modeling on Spoken Response Coherence

Making Decision Like Human: Joint Aspect Category Sentiment Analysis and Rating Prediction with Fine-to-Coarse Reasoning