Performance Evaluation Systems and Metrics for Semantic Communications

ZHENG Yuan,WANG Fengyu,XU Wenjun
DOI: https://doi.org/10.12142/ztetj.202302010
2023-01-01
Abstract:The absence of a unified and generalized performance evaluation system brings challenges to the realization of semantic communication systems. The usages and shortcomings of evaluation methods in existing research are analyzed. Two evaluation metrics, namely the semantic communication efficiency metric Esc and the semantic communication utility metric Usc, are proposed. Esc measures the accuracy of task completion within unit time cost of a communication system under certain resources. Usc measures how close a communication system is to the upper limit of task performance under certain resources. Compared with the existing evaluation methods, the two proposed metrics are of better generality, which can provide guidance for the performance comparison of semantic communication models under different scenarios and information modalities. In addition, the semantic image reconstruction and speech reconstruction are taken as examples to build end-to-end semantic communication simulation models respectively, and the performance of models based on the proposed Esc and Usc metrics is evaluated.
What problem does this paper attempt to address?