Semantic Image Matting

Yanan Sun,Chi-Keung Tang,Yu-Wing Tai

DOI: https://doi.org/10.48550/arXiv.2104.08201

2021-04-17

Abstract:Natural image matting separates the foreground from background in fractional occupancy which can be caused by highly transparent objects, complex foreground (e.g., net or tree), and/or objects containing very fine details (e.g., hairs). Although conventional matting formulation can be applied to all of the above cases, no previous work has attempted to reason the underlying causes of matting due to various foreground semantics. We show how to obtain better alpha mattes by incorporating into our framework semantic classification of matting regions. Specifically, we consider and learn 20 classes of matting patterns, and propose to extend the conventional trimap to semantic trimap. The proposed semantic trimap can be obtained automatically through patch structure analysis within trimap regions. Meanwhile, we learn a multi-class discriminator to regularize the alpha prediction at semantic level, and content-sensitive weights to balance different regularization losses. Experiments on multiple benchmarks show that our method outperforms other methods and has achieved the most competitive state-of-the-art performance. Finally, we contribute a large-scale Semantic Image Matting Dataset with careful consideration of data balancing across different semantic classes.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve several key problems in Natural Image Matting:

1. **Lack of semantic information**: Traditional matting methods mainly rely on low - level image features, such as color or structure, without considering the semantic information of foreground objects. This leads to the failure of traditional methods when dealing with complex foregrounds (such as hair, meshes, trees, etc.).

2. **Boundary and pattern diversity of different foreground objects**: Different types of boundaries or patterns may exist around different foreground objects. For example, a portrait usually has both a fuzzy hair area and a clear body boundary. Existing methods perform poorly when dealing with these diverse boundaries and patterns.

3. **Limitations of the traditional Trimap**: The traditional Trimap only contains information about the foreground, background, and unknown regions, lacking semantic classification of pixels within the unknown regions, which limits the performance of the model in complex scenarios.

To solve the above problems, the paper proposes a new framework to improve the effect of natural image matting by introducing **semantic classification**. Specifically, the main contributions of the paper include:

1. **Introducing semantic information**: Semantic classification is introduced for the first time in the natural image matting task, showing how to use semantic information to achieve the most competitive performance.

2. **Expanding the traditional Trimap**: The **Semantic Trimap** is proposed, which contains multi - channel confidence maps for each unknown region to represent the confidence of different matting categories.

3. **Multi - class discriminator**: A multi - class discriminator is introduced to provide supervision at the semantic level to regularize the alpha prediction results and introduce content - sensitive weights to balance different regularization losses.

4. **Constructing a large - scale dataset**: Contribute the first large - scale and class - balanced **Semantic Image Matting Dataset**, covering a wide range of matting patterns, providing new perspectives and in - depth analysis for future matting research.

Through these innovations, the method proposed in the paper has achieved state - of - the - art performance in multiple benchmark tests, especially when dealing with complex boundaries and patterns of different foreground objects.