Abstract:Text-to-image generation has made remarkable progress with the emergence of diffusion models. However, it is still a difficult task to generate images for street views based on text, mainly because the road topology of street scenes is complex, the traffic status is diverse and the weather condition is various, which makes conventional text-to-image models difficult to deal with. To address these challenges, we propose a novel controllable text-to-image framework, named \textbf{Text2Street}. In the framework, we first introduce the lane-aware road topology generator, which achieves text-to-map generation with the accurate road structure and lane lines armed with the counting adapter, realizing the controllable road topology generation. Then, the position-based object layout generator is proposed to obtain text-to-layout generation through an object-level bounding box diffusion strategy, realizing the controllable traffic object layout generation. Finally, the multiple control image generator is designed to integrate the road topology, object layout and weather description to realize controllable street-view image generation. Extensive experiments show that the proposed approach achieves controllable street-view text-to-image generation and validates the effectiveness of the Text2Street framework for street views.

What problem does this paper attempt to address?

The paper aims to address the controllable text-to-image generation problem in street view imagery, specifically, it attempts to overcome the following three main challenges: 1. Complex road topology structures: Including road structures and topological markings, which may be incomplete in traditional images due to perspective limitations and occlusions, making it difficult for traditional text-to-image models based on diffusion models to handle. 2. Diverse traffic conditions: Such as different numbers of traffic objects, existing models often fail to meet precise quantity requirements when generating a specific number of traffic objects. 3. Various weather conditions: Directly generating images based on weather conditions often leads to blurry or suboptimal results. To address the aforementioned issues, the paper proposes a new controllable text-to-image framework named Text2Street. This framework includes the following key components: - Lane-aware road topology generator: Utilizes text input to create local semantic maps, accurately representing complex road structures and lane lines, achieving controllable road topology generation. - Location-based object layout generator: Generates traffic object layouts from text descriptions through an object-level bounding box diffusion strategy, achieving controllable traffic object layout generation. - Multi-control image generator: Integrates road topology, object layout, and weather descriptions to achieve controllable street view image generation. Experimental results show that Text2Street can effectively generate street view images that match the initial text description's road topology, traffic conditions, and weather conditions. It outperforms existing methods on various evaluation metrics, especially in attribute-level indicators (such as road structure, lane line count, traffic object count, and weather condition accuracy), demonstrating its advantages in fine-grained controllable street view image generation. Additionally, the framework also supports image editing operations, allowing for modifications to the generated images to accommodate different needs.

Text2Street: Controllable Text-to-image Generation for Street Views

Chasing Consistency in Text-to-3D Generation from a Single Image.

How Control Information Influences Multilingual Text Image Generation and Editing?

Control3D: Towards Controllable Text-to-3D Generation

Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformer

Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model

AnyText: Multilingual Visual Text Generation And Editing

Typographic Text Generation with Off-the-Shelf Diffusion Model

GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation

Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion

Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting

From Bird's-Eye to Street View: Crafting Diverse and Condition-Aligned Images with Latent Diffusion Model

Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models

SpaText: Spatio-Textual Representation for Controllable Image Generation

CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis

Text Semantics to Image Generation: A method of building facades design base on Stable Diffusion model

UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models

Generating Urban Road Networks with Conditional Diffusion Models

Controllable Generation with Text-to-Image Diffusion Models: A Survey

Layout Agnostic Scene Text Image Synthesis with Diffusion Models