Abstract:Semantic communication, as a revolutionary communication architecture, is considered a promising novel communication paradigm. Unlike traditional symbol-based error-free communication systems, semantic-based visual communication systems extract, compress, transmit, and reconstruct images at the semantic level. However, widely used image similarity evaluation metrics, whether pixel-based MSE or PSNR or structure-based MS-SSIM, struggle to accurately measure the loss of semantic-level information of the source during system transmission. This presents challenges in evaluating the performance of visual semantic communication systems, especially when comparing them with traditional communication systems. To address this, we propose a semantic evaluation metric -- SeSS (Semantic Similarity Score), based on Scene Graph Generation and graph matching, which shifts the similarity scores between images into semantic-level graph matching scores. Meanwhile, semantic similarity scores for tens of thousands of image pairs are manually annotated to fine-tune the hyperparameters in the graph matching algorithm, aligning the metric more closely with human semantic perception. The performance of the SeSS is tested on different datasets, including (1)images transmitted by traditional and semantic communication systems at different compression rates, (2)images transmitted by traditional and semantic communication systems at different signal-to-noise ratios, (3)images generated by large-scale model with different noise levels introduced, and (4)cases of images subjected to certain special transformations. The experiments demonstrate the effectiveness of SeSS, indicating that the metric can measure the semantic-level differences in semantic-level information of images and can be used for evaluation in visual semantic communication systems.

Measuring Semantic Similarity Between Concepts in Visual Domain.

A Novel Comprehensive Approach for Estimating Concept Semantic Similarity in WordNet

A New Hypred Improved Method for Measuring Concept Semantic Similarity in WordNet.

A Hybrid Semantic Similarity Measurement for Geospatial Entities

Contextual Visual Similarity

Quantitative Characterization of Semantic Gaps for Learning Complexity Estimation and Inference Model Selection

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Measuring the Semantic Relatedness Between Images Using Social Tags.

Semantic Discriminative Metric Learning for Image Similarity Measurement

An integrated semantic-based approach in concept based video retrieval

Semantic Similarity Score for Measuring Visual Similarity at Semantic Level

What Are The High-Level Concepts With Small Semantic Gaps?

A Semantic Context Model For Automatic Image Annotation

Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding

A Similarity Metric in Image Searching.

Video Similarity Measurement

Efficient Multimedia Similarity Measurement Using Similar Elements

Modeling Image Data for Effective Indexing and Retrieval in Large General Image Databases.

Measuring semantic relatedness between Flickr images: from a social tag based view.

Video Semantic Concept Detection Using Multi-Modality Subspace Correlation Propagation

Understanding the spatial dimension of natural language by measuring the spatial semantic similarity of words through a scalable geospatial context window