Abstract:The emergence of Large Language Models (LLMs) and multimodal foundation models (FMs) has generated heightened interest in their applications that integrate vision and language. This paper investigates the capabilities of ChatGPT-4V and Gemini Pro for Street View Imagery, Built Environment, and Interior by evaluating their performance across various tasks. The assessments include street furniture identification, pedestrian and car counts, and road width measurement in Street View Imagery; building function classification, building age analysis, building height analysis, and building structure classification in the Built Environment; and interior room classification, interior design style analysis, interior furniture counts, and interior length measurement in Interior. The results reveal proficiency in length measurement, style analysis, question answering, and basic image understanding, but highlight limitations in detailed recognition and counting tasks. While zero-shot learning shows potential, performance varies depending on the problem domains and image complexities. This study provides new insights into the strengths and weaknesses of multimodal foundation models for practical challenges in Street View Imagery, Built Environment, and Interior. Overall, the findings demonstrate foundational multimodal intelligence, emphasizing the potential of FMs to drive forward interdisciplinary applications at the intersection of computer vision and language.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the application capabilities and limitations of multimodal foundation models (FMs) in street - view images, built environments, and indoor scenes. Specifically, researchers test the performance of these models through a series of tasks, including but not limited to: 1. **Street - view images**: - **Street furniture recognition**: Recognize street facilities such as benches and trash cans. - **Pedestrian counting**: Accurately count the number of pedestrians. - **Vehicle counting**: Count the number of different types of vehicles. - **Road width measurement**: Estimate the width of the road. 2. **Built environment**: - **Building function classification**: Judge its use (such as residential, commercial, educational, etc.) according to the appearance characteristics of the building. - **Building age analysis**: Estimate the construction age of the building. - **Building height analysis**: Measure the height of the building. - **Building structure classification**: Identify the structural type of the building (such as frame structure, brick - concrete structure, etc.). 3. **Indoor scenes**: - **Room type classification**: Identify the function of the room (such as living room, bedroom, kitchen, etc.). - **Interior design style analysis**: Judge the interior decoration style (such as modern, classical, etc.). - **Indoor furniture counting**: Count the number of indoor furniture. - **Indoor length measurement**: Measure certain specific dimensions indoors. The paper evaluates the performance of two multimodal foundation models, ChatGPT - 4V and Gemini Pro, in the above tasks, revealing their advantages and disadvantages in handling tasks that combine vision and language. The research results show that these models perform well in length measurement, style analysis, question - answering, and basic image understanding, but have limitations in detailed identification and counting tasks. In addition, zero - shot learning shows potential, but its performance varies depending on the problem domain and image complexity. Overall, this research provides new insights into the performance of multimodal foundation models in practical applications and emphasizes the potential of these models in promoting interdisciplinary applications in the cross - field of computer vision and language.

Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery