Synthetic Instance Segmentation from Semantic Image Segmentation Masks

Yuchen Shen,Dong Zhang,Zhao Zhang,Liyong Fu,Qiaolin Ye

2024-10-08

Abstract:In recent years, instance segmentation has garnered significant attention across various applications. However, training a fully-supervised instance segmentation model requires costly both instance-level and pixel-level annotations. In contrast, weakly-supervised instance segmentation methods, such as those using image-level class labels or point labels, often struggle to satisfy the accuracy and recall requirements of practical scenarios. In this paper, we propose a novel paradigm called Synthetic Instance Segmentation (SISeg). SISeg achieves instance segmentation results by leveraging image masks generated by existing semantic segmentation models, and it is highly efficient as we do not require additional training for semantic segmentation or the use of instance-level image annotations. In other words, the proposed model does not need extra manpower or higher computational expenses. Specifically, we first obtain a semantic segmentation mask of the input image via an existent semantic segmentation model. Then, we calculate a displacement field vector for each pixel based on the segmentation mask, which can indicate representations belonging to the same class but different instances, i.e., obtaining the instance-level object information. Finally, the instance segmentation results are refined by a learnable category-agnostic object boundary branch. Extensive experimental results on two challenging datasets highlight the effectiveness of SISeg in achieving competitive results when compared to state-of-the-art methods, especially fully-supervised methods. The code will be released at: SISeg

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of reducing the dependency on instance-level annotated data in instance segmentation tasks, thereby improving the efficiency of model training and reducing costs. Specifically, existing fully supervised instance segmentation methods require a large amount of instance-level and pixel-level annotated data, which is not only time-consuming but also costly. Although weakly supervised instance segmentation methods can reduce annotation costs, they often fail to meet the accuracy requirements in practical applications.

To this end, the paper proposes a new instance segmentation method—Synthetic Instance Segmentation (SISeg). SISeg achieves efficient instance segmentation by utilizing semantic segmentation masks generated by existing semantic segmentation models, combined with a Displacement Field Detection Module and a Category Boundary Refinement Module. This method does not require additional instance-level annotated data or retraining of the semantic segmentation model, thereby significantly improving efficiency while ensuring accuracy.

### Main Contributions:
1. **Proposed a new instance segmentation method SISeg**: This method is based on existing semantic segmentation models and can generate high-quality instance segmentation results without additional training.
2. **Efficiency**: SISeg does not rely on instance-level annotated data and only uses pixel-level annotations, greatly reducing annotation costs and computational resource requirements.
3. **Superior Performance**: Experimental results show that SISeg achieves performance comparable to existing fully supervised instance segmentation methods on the challenging PASCAL VOC 2012 and ADE20K datasets.

### Method Overview:
1. **Displacement Field Detection Module (DFM)**: Distinguishes different instances of the same category by predicting the displacement vector of each pixel.
2. **Category Boundary Refinement Module (CBR)**: Generates clear boundary information by learning the semantic similarity between pixel pairs, further optimizing the segmentation results.
3. **Semantic-Aware Propagation Module**: Enhances segmentation accuracy, especially at object boundaries, by diffusing high-similarity regions using a random walk algorithm.

### Experimental Validation:
The paper conducts extensive experiments on the PASCAL VOC 2012 and ADE20K datasets to validate the effectiveness and competitiveness of SISeg. The experimental results demonstrate that SISeg not only achieves comparable accuracy to existing methods but also has significant advantages in terms of efficiency and cost.