Abstract:Semantic segmentation datasets often exhibit two types of imbalance: \textit{class imbalance}, where some classes appear more frequently than others and \textit{size imbalance}, where some objects occupy more pixels than others. This causes traditional evaluation metrics to be biased towards \textit{majority classes} (e.g. overall pixel-wise accuracy) and \textit{large objects} (e.g. mean pixel-wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mIoUs along with corresponding worst-case metrics, thereby offering a more holistic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, we undertake an extensive benchmark study, where we train and evaluate 15 modern neural networks with the proposed metrics on 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead to best practices in optimizing fine-grained metrics. The code is available at \href{<a class="link-external link-https" href="https://github.com/zifuwanggg/JDTLosses" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/zifuwanggg/JDTLosses" rel="external noopener nofollow">this https URL</a>}.

Misclassified Evaluation Metrics for 3D Mesh Segmentation

Accuracy Evaluation of Segmentation for High Resolution Imagery and 3D Laser Point Cloud Data

Objective Performance Evaluation of Video Segmentation Algorithms with Ground-Truth

A Novel Quality Evaluating Method for Over-Segmentation Approaches Using Real-Time Boundary Information

3D mesh segmentation using a two-stage merging strategy

Mesh Segmentation with Concavity-Aware Fields

Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

A Novel Mesh Quad-segmentation Algorithm

Segmentation metric misinterpretations in bioimage analysis

A novel segmentation algorithm for complex 3D mesh model in computer vision.

Metrics Revolutions: Groundbreaking Insights into the Implementation of Metrics for Biomedical Image Segmentation

Accuracy Assessment Measures for Object-Based Image Segmentation Goodness

Fine-grained Metrics for Point Cloud Semantic Segmentation

CEFM - A Heuristic Mesh Segmentation Method Based on Convexity Estimation and Fast Marching.

Objective Image Quality Measures and Their Applications in Segmentation Evaluation

Evaluating Multi-Class Segmentation Errors with Anatomical Priors.

Geometric Segmentation of 3D Scanned Surfaces for Multi-Sensor Coordinate Metrology

Towards a guideline for evaluation metrics in medical image segmentation

Segmentation Quality Evaluation Using Region-Based Precision and Recall Measures for Remote Sensing Images

Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union

A Fast and Effective Method for Unsupervised Segmentation Evaluation of Remote Sensing Images