Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems

Sanjita Prajapati,Tanu Singh,Chinmay Hegde,Pranamesh Chakraborty
2024-09-04
Abstract:Recent developments in vision language models (VLM) have shown great potential for diverse applications related to image understanding. In this study, we have explored state-of-the-art VLM models for vision-based transportation engineering tasks such as image classification and object detection. The image classification task involves congestion detection and crack identification, whereas, for object detection, helmet violations were identified. We have applied open-source models such as CLIP, BLIP, OWL-ViT, Llava-Next, and closed-source GPT-4o to evaluate the performance of these state-of-the-art VLM models to harness the capabilities of language understanding for vision-based transportation tasks. These tasks were performed by applying zero-shot prompting to the VLM models, as zero-shot prompting involves performing tasks without any training on those tasks. It eliminates the need for annotated datasets or fine-tuning for specific tasks. Though these models gave comparative results with benchmark Convolutional Neural Networks (CNN) models in the image classification tasks, for object localization tasks, it still needs improvement. Therefore, this study provides a comprehensive evaluation of the state-of-the-art VLM models highlighting the advantages and limitations of the models, which can be taken as the baseline for future improvement and wide-scale implementation.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The paper attempts to address the problem of evaluating and comparing the performance of Visual Language Models (VLM) in traffic engineering tasks. Specifically, the researchers focused on the following aspects: 1. **Image Classification Tasks**: - **Congestion Detection**: Identifying whether highway lanes are congested from surveillance cameras and classifying images as congested or non-congested. - **Crack Detection**: Identifying whether there are cracks on the road surface and classifying images as cracked or non-cracked. 2. **Object Detection Tasks**: - **Helmet Violation Detection**: Identifying whether motorcycle riders are wearing helmets, which is an important traffic safety issue. The goal is to detect two categories: "Helmet" (wearing a helmet) and "No-Helmet" (not wearing a helmet). The researchers selected several state-of-the-art Visual Language Models (such as CLIP, BLIP, OWL-ViT, Llava-Next, and GPT-4o) to evaluate their performance on the above tasks through zero-shot prompting. Zero-shot prompting refers to the model performing tasks without being specifically trained for them, eliminating the need for labeled datasets or fine-tuning. Through these tasks, the researchers aim to comprehensively assess the strengths and limitations of these VLM models, providing a baseline for future improvements and broader applications.