Semantic Segmentation of Street Scenes Using Disparity Information

Hanwen Hu,Xu Zhao

DOI: https://doi.org/10.1007/978-3-030-34120-6_14

2019-01-01

Abstract:In this work, we address the task of semantic segmentation in street scenes. Recent approaches based on convolutional neural networks have shown excellent results on several semantic segmentation benchmarks. Most of them, however, only exploit RGB information. Due to the development of stereo matching algorithms, disparity maps can be more easily acquired. Structural information encoded in disparity can be treated as supplementary information of RGB images, which is expected to boost performance. Therefore, in this work we propose to fuse disparity information in street scene understanding task. And we design four methods to incorporate disparity information into semantic segmentation framework. They are summation, multiplication, concatenation and channel concatenation. Besides, disparity map can be utilized as ground truth of a regression task, guiding the learning of semantic segmentation as a loss term. Comprehensive experiments on KITTI and Cityscapes datasets show that each method can achieve performance improvement. The experimental results validate the effectiveness of disparity information to street scene semantic segmentation tasks.