Abstract:This paper explores the application of the GPT-4V(ision) large visual language model to autonomous driving in mining environments, where traditional systems often falter in understanding intentions and making accurate decisions during emergencies. GPT-4V introduces capabilities for visual question answering and complex scene comprehension, addressing challenges in these specialized settings.Our evaluation focuses on its proficiency in scene understanding, reasoning, and driving functions, with specific tests on its ability to recognize and interpret elements such as pedestrians, various vehicles, and traffic devices. While GPT-4V showed robust comprehension and decision-making skills, it faced difficulties in accurately identifying specific vehicle types and managing dynamic interactions. Despite these challenges, its effective navigation and strategic decision-making demonstrate its potential as a reliable agent for autonomous driving in the complex conditions of mining environments, highlighting its adaptability and operational viability in industrial settings.

What problem does this paper attempt to address?

The paper primarily explores how to utilize GPT-4V (a large visual language model) to implement autonomous driving technology in mining environments. Addressing the limitations of traditional autonomous driving systems in understanding scene intentions and making accurate decisions in emergency situations, GPT-4V introduces capabilities for visual question answering and complex scene understanding. The paper evaluates the performance of GPT-4V in the following three key aspects through a series of experiments: 1. **Scene Understanding**: Evaluating GPT-4V's ability to recognize and interpret elements such as pedestrians, various types of vehicles, machinery, ore piles, and traffic signals in mining environments. While GPT-4V excels in many areas, it faces difficulties in accurately identifying specific types of vehicles and managing dynamic interactions. 2. **Reasoning Ability**: Testing GPT-4V's ability to understand the environment and formulate response strategies in emergency or extreme events, as well as its understanding of the behavior of other vehicles over time. GPT-4V demonstrates strong environmental understanding and decision-making capabilities, but still faces challenges in some specific tasks. 3. **Performance as a Driver**: Evaluating GPT-4V's ability to perform driving tasks such as turning, overtaking, path planning, parking, and lane changing. Through these tests, researchers can analyze GPT-4V's performance in complex driving situations and its ability to make real-time strategic decisions. Despite some limitations, such as difficulty in accurately identifying certain vehicle types and precisely judging vehicle speed and direction, the potential shown by GPT-4V in navigation and strategic decision-making suggests that it could become a reliable autonomous driving agent in complex mining conditions.

GPT-4V Explorations: Mining Autonomous Driving

On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving

GPT-4V as Traffic Assistant: An In-depth Look at Vision Language Model on Complex Traffic Events

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models

An Early Evaluation of GPT-4V(ision)

GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction

Embodied Intelligence in Mining: Leveraging Multi-modal Large Language Model for Autonomous Driving in Mines

Map Reading and Analysis with GPT-4V(ision)

Embodied Intelligence in Mining: Leveraging Multi-Modal Large Language Models for Autonomous Driving in Mines

DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT

DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers

On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applications

A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks

High-Precision Positioning, Perception and Safe Navigation for Automated Heavy-duty Mining Trucks

Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles

GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?

MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation

GPT-Driver: Learning to Drive with GPT