Abstract:The pursuit of autonomous driving technology hinges on the sophisticated integration of perception, decision-making, and control systems. Traditional approaches, both data-driven and rule-based, have been hindered by their inability to grasp the nuance of complex driving environments and the intentions of other road users. This has been a significant bottleneck, particularly in the development of common sense reasoning and nuanced scene understanding necessary for safe and reliable autonomous driving. The advent of Visual Language Models (VLM) represents a novel frontier in realizing fully autonomous vehicle driving. This report provides an exhaustive evaluation of the latest state-of-the-art VLM, GPT-4V(ision), and its application in autonomous driving scenarios. We explore the model's abilities to understand and reason about driving scenes, make decisions, and ultimately act in the capacity of a driver. Our comprehensive tests span from basic scene recognition to complex causal reasoning and real-time decision-making under varying conditions. Our findings reveal that GPT-4V demonstrates superior performance in scene understanding and causal reasoning compared to existing autonomous systems. It showcases the potential to handle out-of-distribution scenarios, recognize intentions, and make informed decisions in real driving contexts. However, challenges remain, particularly in direction discernment, traffic light recognition, vision grounding, and spatial reasoning tasks. These limitations underscore the need for further research and development. Project is now available on GitHub for interested parties to access and utilize: \url{<a class="link-external link-https" href="https://github.com/PJLab-ADG/GPT4V-AD-Exploration" rel="external noopener nofollow">this https URL</a>}

SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving

VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision

DriveLM: Driving with Graph Visual Question Answering

Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

LLM4Drive: A Survey of Large Language Models for Autonomous Driving

Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Simulation, and Real-Vehicle Experiment

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

Vision Language Models in Autonomous Driving: A Survey and Outlook

DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving

Empowering Autonomous Driving with Large Language Models: A Safety Perspective

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

Drive Like a Human: Rethinking Autonomous Driving with Large Language Models

VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes

VLP: Vision Language Planning for Autonomous Driving

A Survey on Multimodal Large Language Models for Autonomous Driving

On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving

OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning

V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

A Language Agent for Autonomous Driving