Fine-tuning vision foundation model for crack segmentation in civil infrastructures

Kang Ge,Chen Wang,Yutao Guo,Yansong Tang,Zhenzhong Hu,Hongbing Chen

2024-04-23

Abstract:Large-scale foundation models have become the mainstream deep learning method, while in civil engineering, the scale of AI models is strictly limited. In this work, a vision foundation model is introduced for crack segmentation. Two parameter-efficient fine-tuning methods, adapter and low-rank adaptation, are adopted to fine-tune the foundation model in semantic segmentation: the Segment Anything Model (SAM). The fine-tuned CrackSAM shows excellent performance on different scenes and materials. To test the zero-shot performance of the proposed method, two unique datasets related to road and exterior wall cracks are collected, annotated and open-sourced, for a total of 810 images. Comparative experiments are conducted with twelve mature semantic segmentation models. On datasets with artificial noise and previously unseen datasets, the performance of CrackSAM far exceeds that of all state-of-the-art models. CrackSAM exhibits remarkable superiority, particularly under challenging conditions such as dim lighting, shadows, road markings, construction joints, and other interference factors. These cross-scenario results demonstrate the outstanding zero-shot capability of foundation models and provide new ideas for developing vision models in civil engineering.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of crack segmentation in civil engineering structures. Traditional manual crack detection methods are costly, highly subjective, and inefficient. In recent years, although computer vision (CV) technology has made significant progress in crack segmentation tasks, existing crack segmentation models have poor generalization capabilities across different datasets, especially lacking robustness in complex environments. Additionally, crack segmentation is a highly class-imbalanced problem and is affected by various interfering factors such as insufficient lighting, shadows, and occlusions.

To tackle these issues, the authors introduce large-scale foundational models and explore their application in crack segmentation. Specifically, the paper employs two parameter-efficient fine-tuning methods—Adapter and Low-Rank Adaptation (LoRA)—to fine-tune the foundational model Segment Anything Model (SAM) to enhance its zero-shot generalization capability in different scenarios. The paper also collects two unique datasets (Road420 and Facade390), containing pavement and facade cracks, and incorporates artificial noise to simulate complex conditions in real-world engineering. Experimental results show that the fine-tuned CrackSAM model performs excellently under various interference conditions, significantly outperforming other existing models in zero-shot testing.

In summary, by introducing large-scale foundational models and efficient fine-tuning strategies, this paper aims to enhance the robustness and generalization capabilities of crack segmentation models in complex environments, thereby advancing their practical application.