WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather

Blake Gella,Howard Zhang,Rishi Upadhyay,Tiffany Chang,Matthew Waliman,Yunhao Ba,Alex Wong,Achuta Kadambi

2023-12-15

Abstract:The introduction of large, foundational models to computer vision has led to drastically improved performance on the task of semantic segmentation. However, these existing methods exhibit a large performance drop when testing on images degraded by weather conditions such as rain, fog, or snow. We introduce a general paired-training method that can be applied to all current foundational model architectures that leads to improved performance on images in adverse weather conditions. To this end, we create the WeatherProof Dataset, the first semantic segmentation dataset with accurate clear and adverse weather image pairs, which not only enables our new training paradigm, but also improves the evaluation of the performance gap between clear and degraded segmentation. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data. With this knowledge, we propose a training pipeline which accentuates the advantages of paired-data training using consistency losses and language guidance, which leads to performance improvements by up to 18.4% as compared to standard training procedures.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily addresses the issue of performance degradation in semantic segmentation tasks in computer vision under adverse weather conditions. Specifically, although large foundational models have achieved significant results on standard datasets, their performance drops significantly when dealing with images affected by weather conditions such as rain, fog, or snow.

To address this issue, the authors propose the following contributions:

1. **WeatherProof Dataset**: This is the first dataset containing high-quality semantic segmentation labels, including pairs of clear images and images under adverse weather conditions. This dataset allows researchers to conduct paired training and more accurately assess the performance gap of models under different weather conditions.

2. **Paired Training Method**: By leveraging the pairing relationship between clear images and adverse weather images in the WeatherProof dataset, researchers can design a new training method. This method allows the model to focus separately on adapting to new scenes and adapting to weather changes within the same scene, thereby improving the model's performance under adverse weather conditions.

3. **Consistency Loss**: To further enhance the model's robustness under adverse weather conditions, the authors introduce Feature Consistency Loss (FCL) and Output Consistency Loss (OCL). These loss functions encourage the model to learn latent representations that are less affected by weather.

4. **CLIP Injection Layer**: Additionally, the paper proposes a CLIP Injection Layer, which uses natural language descriptions to guide the model in identifying the components of weather conditions in images, thereby helping the model better cope with adverse weather conditions.

Through the above methods, the paper demonstrates that the proposed paired training method can significantly improve the model's mean Intersection over Union (mIoU) under adverse weather conditions, with an increase of up to 18.4%. This indicates that by utilizing accurate paired data and targeted technical improvements, the semantic segmentation capability of existing foundational models under adverse weather conditions can be effectively enhanced.