Abstract:With the open-sourcing of text-to-image models (T2I) such as stable diffusion (SD) and stable diffusion XL (SD-XL), there is an influx of models fine-tuned in specific domains based on the open-source SD model, such as in anime, character portraits, etc. However, there are few specialized models in certain domains, such as interior design, which is attributed to the complex textual descriptions and detailed visual elements inherent in design, alongside the necessity for adaptable resolution. Therefore, text-to-image models for interior design are required to have outstanding prompt-following capabilities, as well as iterative collaboration with design professionals to achieve the desired outcome. In this paper, we collect and optimize text-image data in the design field and continue training in both English and Chinese on the basis of the open-source CLIP model. We also proposed a fine-tuning strategy with curriculum learning and reinforcement learning from CLIP feedback to enhance the prompt-following capabilities of our approach so as to improve the quality of image generation. The experimental results on the collected dataset demonstrate the effectiveness of the proposed approach, which achieves impressive results and outperforms strong baselines.

What problem does this paper attempt to address?

The paper aims to address the application of Text-to-Image (T2I) models in the field of interior design. Specifically, the paper investigates the following points: 1. **Limitations of existing T2I models**: Although existing T2I models such as Stable Diffusion (SD) and Stable Diffusion XL (SD-XL) have made progress in general media generation, they fall short when dealing with complex textual descriptions specific to the field of interior design. This is mainly because these models struggle to handle the rich and complex vocabulary unique to interior design, resulting in generated images that lack the precision and detail required for professional design. 2. **Proposed new model iDesigner**: To address the above issues, the paper proposes a new model named iDesigner. This model is specifically optimized for the field of interior design and can generate visually rich and contextually accurate interior design images directly from descriptive text prompts. 3. **Key technical methods**: - **Curriculum Learning**: The model is trained by gradually increasing the complexity of tasks, starting with low-resolution images and progressively moving to high-resolution image generation. This ensures that the model can gradually master the ability to handle everything from basic design principles to fine details. - **Reinforcement Learning with CLIP Feedback (RLCF)**: A domain-specific trained CLIP model is used as a feedback mechanism. Through reinforcement learning, the model continuously optimizes its understanding and response to text prompts, improving the quality and accuracy of the generated images. - **Data Recaptioning**: To improve data quality, GPT-3.5 is used to rewrite the original labels, generating more detailed and accurate descriptive text, further enhancing the quality of image generation. 4. **Experimental results**: - Experimental results on the collected dataset show that iDesigner performs excellently on multiple evaluation metrics, including CLIP similarity, Inception score, and Fréchet Inception distance, significantly outperforming other baseline models such as DALL-E 3 and SD-XL. - Human preference evaluations also indicate that participants generally consider the images generated by iDesigner to be of higher quality, more aligned with the text descriptions, and possessing greater aesthetic appeal. In summary, by proposing the iDesigner model and its series of innovative technical methods, this paper effectively addresses the limitations of existing T2I models in the application of interior design, demonstrating the strong potential for generating high-quality images in this specific field.

iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Design

RoomDiffusion: A Specialized Diffusion Model in the Interior Design Industry

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

TextDiffuser: Diffusion Models as Text Painters

Generating Interior Design from Text: A New Diffusion Model-Based Method for Efficient Creative Design

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models

Prompt-Free Diffusion: Taking "text" out of Text-to-Image Diffusion Models

DiffDesign: Controllable Diffusion with Meta Prior for Efficient Interior Design Generation

Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following

Integrating aesthetics and efficiency: AI-driven diffusion models for visually pleasing interior design generation

DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing

TextCraftor: Your Text Encoder Can be Image Quality Controller

Design Booster: A Text-Guided Diffusion Model for Image Translation with Spatial Layout Preservation

IconDM: Text-Guided Icon Set Expansion Using Diffusion Models

ColorEdit: Training-free Image-Guided Color editing with diffusion model

ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models

Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models

InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists

DreamDistribution: Prompt Distribution Learning for Text-to-Image Diffusion Models

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models