Abstract:The pursuit of autonomous driving technology hinges on the sophisticated integration of perception, decision-making, and control systems. Traditional approaches, both data-driven and rule-based, have been hindered by their inability to grasp the nuance of complex driving environments and the intentions of other road users. This has been a significant bottleneck, particularly in the development of common sense reasoning and nuanced scene understanding necessary for safe and reliable autonomous driving. The advent of Visual Language Models (VLM) represents a novel frontier in realizing fully autonomous vehicle driving. This report provides an exhaustive evaluation of the latest state-of-the-art VLM, GPT-4V(ision), and its application in autonomous driving scenarios. We explore the model's abilities to understand and reason about driving scenes, make decisions, and ultimately act in the capacity of a driver. Our comprehensive tests span from basic scene recognition to complex causal reasoning and real-time decision-making under varying conditions. Our findings reveal that GPT-4V demonstrates superior performance in scene understanding and causal reasoning compared to existing autonomous systems. It showcases the potential to handle out-of-distribution scenarios, recognize intentions, and make informed decisions in real driving contexts. However, challenges remain, particularly in direction discernment, traffic light recognition, vision grounding, and spatial reasoning tasks. These limitations underscore the need for further research and development. Project is now available on GitHub for interested parties to access and utilize: \url{<a class="link-external link-https" href="https://github.com/PJLab-ADG/GPT4V-AD-Exploration" rel="external noopener nofollow">this https URL</a>}

VistaGPT: Generative Parallel Transformers for Vehicles With Intelligent Systems for Transport Automation

HiVeGPT: Human-Machine-Augmented Intelligent Vehicles with Generative Pre-Trained Transformer

Expression of Concern For: HiVeGPT: Human-Machine-Augmented Intelligent Vehicles with Generative Pre-Trained Transformer

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability

Federated Vehicular Transformers and Their Federations: Privacy-Preserving Computing and Cooperation for Autonomous Driving

Parallel Driving with Big Models and Foundation Intelligence in Cyber-Physical-Social Spaces

Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses

On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving

Generative AI-enabled Vehicular Networks: Fundamentals, Framework, and Case Study

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

VistaScenario: Interaction Scenario Engineering for Vehicles with Intelligent Systems for Transport Automation

TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation

Futuristic Intelligent Transportation System

Hybrid-Generative Diffusion Models for Attack-Oriented Twin Migration in Vehicular Metaverses

Enhancing Vehicular Networks with Generative AI: Opportunities and Challenges

HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

Parallel Vision for Long-Tail Regularization: Initial Results from IVFC Autonomous Driving Testing