How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model

Yixin Zhang,Shen Zhao,Hanxue Gu,Maciej A. Mazurowski

2023-12-21

Abstract:Deep neural networks (DNNs) have been deployed for many image segmentation tasks and achieved outstanding performance. However, preparing a dataset for training segmentation DNNs is laborious and costly since typically pixel-level annotations are provided for each object of interest. To alleviate this issue, one can provide only weak labels such as bounding boxes or scribbles, or less accurate (noisy) annotations of the objects. These are significantly faster to generate and thus result in more annotated images given the same time budget. However, the reduction in quality might negatively affect the segmentation performance of the resulting model. In this study, we perform a thorough cost-effectiveness evaluation of several weak and noisy labels. We considered 11 variants of annotation strategies and 4 datasets. We conclude that the common practice of accurately outlining the objects of interest is virtually never the optimal approach when the annotation time is limited, even if notable annotation time is available (10s of hours). Annotation approaches that stood out in such scenarios were (1) contour-based annotation with rough continuous traces, (2) polygon-based annotation with few vertices, and (3) box annotations combined with the Segment Anything Model (SAM). In situations where unlimited annotation time was available, precise annotations still lead to the highest segmentation model performance.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of how to efficiently annotate in image segmentation tasks. Specifically, it explores how to choose the most effective annotation method to train deep learning segmentation models under a limited annotation budget (time and money). Traditional pixel-level annotations, while precise, are time-consuming and costly. Therefore, the paper investigates the cost-effectiveness of weak annotations (such as bounding boxes, points, scribbles, etc.) and noisy annotations (such as rough contours) and evaluates these different annotation methods. The main contributions of the study include: 1. **Cost-effectiveness analysis**: Through empirical research, 11 different annotation strategies and their performance on 4 datasets were evaluated. 2. **Optimal annotation methods**: It was found that when annotation time is limited, common precise annotations are not the best choice. Instead, rough contour annotations, polygon annotations with fewer vertices, and bounding box annotations combined with the Segment Anything Model (SAM) showed higher cost-effectiveness. 3. **Performance in different scenarios**: In scenarios with unlimited annotation time, precise annotations still yield the highest segmentation model performance; however, in limited time scenarios, the aforementioned imprecise annotation methods are more effective. The paper provides specific guidance for researchers developing segmentation models on how to efficiently create annotations within a limited time through comparative experiments of different annotation methods.

How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model

Coarse-to-Fine Annotation Enrichment for Semantic Segmentation Learning

An Annotation Sparsification Strategy for 3D Medical Image Segmentation via Representative Selection and Self-Training

Block Annotation: Better Image Annotation for Semantic Segmentation with Sub-Image Decomposition

Accelerating the creation of instance segmentation training sets through bounding box annotation

Annotation quality vs. quantity for deep-learned medical image segmentation

How to select slices for annotation to train best-performing deep learning segmentation models for cross-sectional medical images?

Annotation-Efficient Learning for Medical Image Segmentation Based on Noisy Pseudo Labels and Adversarial Learning

Toward high-quality pseudo masks from noisy or weak annotations for robust medical image segmentation

A sparse annotation strategy based on attention-guided active learning for 3D medical image segmentation

Sparse annotation strategies for segmentation of short axis cardiac MRI

Learning to Segment from Noisy Annotations: A Spatial Correction Approach

Robustness study of noisy annotation in deep learning based medical image segmentation

Optimal Scale of Hierarchical Image Segmentation with Scribbles Guidance for Weakly Supervised Semantic Segmentation

ZScribbleSeg: Zen and the Art of Scribble Supervised Medical Image Segmentation

Noisy Boundaries: Lemon or Lemonade for Semi-supervised Instance Segmentation?

Medical Image Segmentation with SAM-generated Annotations

Elucidating Meta-Structures of Noisy Labels in Semantic Segmentation by Deep Neural Networks

ESA: Annotation-Efficient Active Learning for Semantic Segmentation

Reducing the Annotation Effort for Video Object Segmentation Datasets

FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models