Abstract:Recent advancements in large vision language models (VLMs) tailored for autonomous driving (AD) have shown strong scene understanding and reasoning capabilities, making them undeniable candidates for end-to-end driving systems. However, limited work exists on studying the trustworthiness of DriveVLMs -- a critical factor that directly impacts public transportation safety. In this paper, we introduce AutoTrust, a comprehensive trustworthiness benchmark for large vision-language models in autonomous driving (DriveVLMs), considering diverse perspectives -- including trustfulness, safety, robustness, privacy, and fairness. We constructed the largest visual question-answering dataset for investigating trustworthiness issues in driving scenarios, comprising over 10k unique scenes and 18k queries. We evaluated six publicly available VLMs, spanning from generalist to specialist, from open-source to commercial models. Our exhaustive evaluations have unveiled previously undiscovered vulnerabilities of DriveVLMs to trustworthiness threats. Specifically, we found that the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperform specialized models fine-tuned for driving in terms of overall trustworthiness. DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosing sensitive information. Additionally, both generalist and specialist VLMs remain susceptible to adversarial attacks and struggle to ensure unbiased decision-making across diverse environments and populations. Our findings call for immediate and decisive action to address the trustworthiness of DriveVLMs -- an issue of critical importance to public safety and the welfare of all citizens relying on autonomous transportation systems. Our benchmark is publicly available at \url{<a class="link-external link-https" href="https://github.com/taco-group/AutoTrust" rel="external noopener nofollow">this https URL</a>}, and the leaderboard is released at \url{<a class="link-external link-https" href="https://taco-group.github.io/AutoTrust/" rel="external noopener nofollow">this https URL</a>}.

A Trusted Supervision Paradigm for Autonomous Driving Based on Multimodal Data Authentication

Driving Behaviour Style Study with a Hybrid Deep Learning Framework Based on GPS Data

A Multimodal Data-Driven Approach for Driving Risk Assessment

Trust Calibration Through Perceptual and Predictive Information of the External Context in Autonomous Vehicle

Human-Road Dual Trust Mechanism in Adaptive Distributed Shared Control Framework under Lane Keeping Scenario

Smart and Secure CAV Networks Empowered by AI-Enabled Blockchain: The Next Frontier for Intelligent Safe Driving Assessment

Trust-Evaluation-Based Intrusion Detection and Reinforcement Learning in Autonomous Driving

Challenges and Solutions in Autonomous Driving: A Blockchain Approach

Trusted and Collaborative Data Sharing with Quality Awareness in Autonomous Driving

AutoSup: Driving Behaviors Recognition Supervision Via Incentive Mechanism

SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models

A Data-Informed Analysis of Scalable Supervision for Safety in Autonomous Vehicle Fleets

Trust-Based Shared Control of Human-Vehicle System Using Model Free Adaptive Dynamic Programming

AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving

A Blockchain-Based Multidomain Authentication Scheme for Conditional Privacy Preserving in Vehicular Ad-Hoc Network

A Blockchain-based Multi-domain Authentication Scheme for Conditional Privacy Preserving in Vehicular Ad-hoc Network

Secure Data Sharing for Autonomous Vehicles in Mobile Blockchain Networks

Reputation-Driven Asynchronous Federated Learning for Enhanced Trajectory Prediction with Blockchain

Towards Safe Autonomy in Hybrid Traffic: Detecting Unpredictable Abnormal Behaviors of Human Drivers via Information Sharing

A Case Study of Trust on Autonomous Driving

An Incentive Mechanism of Incorporating Supervision Game for Federated Learning in Autonomous Driving