Beyond Segmentation: Road Network Generation with Multi-Modal LLMs

Sumedh Rasal,Sanjay Kumar Boddhu
DOI: https://doi.org/10.48550/arXiv.2310.09755
2023-10-15
Abstract:This paper introduces an innovative approach to road network generation through the utilization of a multi-modal Large Language Model (LLM). Our model is specifically designed to process aerial images of road layouts and produce detailed, navigable road networks within the input images. The core innovation of our system lies in the unique training methodology employed for the large language model to generate road networks as its output. This approach draws inspiration from the BLIP-2 architecture <a class="link-https" data-arxiv-id="2301.12597" href="https://arxiv.org/abs/2301.12597">arXiv:2301.12597</a>, leveraging pre-trained frozen image encoders and large language models to create a versatile multi-modal LLM. Our work also offers an alternative to the reasoning segmentation method proposed in the LISA paper <a class="link-https" data-arxiv-id="2308.00692" href="https://arxiv.org/abs/2308.00692">arXiv:2308.00692</a>. By training the large language model with our approach, the necessity for generating binary segmentation masks, as suggested in the LISA paper <a class="link-https" data-arxiv-id="2308.00692" href="https://arxiv.org/abs/2308.00692">arXiv:2308.00692</a>, is effectively eliminated. Experimental results underscore the efficacy of our multi-modal LLM in providing precise and valuable navigational guidance. This research represents a significant stride in bolstering autonomous navigation systems, especially in road network scenarios, where accurate guidance is of paramount importance.
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
The paper aims to address the problem of generating navigable road networks from aerial images. Specifically, the authors propose an innovative approach that utilizes a multimodal large language model (LLM) to process aerial images and generate detailed and navigable road networks. The core innovation of this method lies in its unique training approach, which creates a versatile multimodal LLM by pre-training frozen image encoders and large language models. Compared to previous research, such as the segmentation method proposed in the LISA paper, this method eliminates the need to generate binary segmentation masks. Experimental results show that this multimodal LLM has significant effectiveness in providing precise and valuable navigation guidance, which is particularly important for enhancing the accuracy of autonomous driving systems in road network scenarios. In this way, the researchers demonstrate the potential of large language models in map-making, which is expected to advance the realization of autonomous navigation.