Abstract:Humans apprehend the world through various sensory modalities, yet language is their predominant communication channel. Machine learning systems need to draw on the same multimodal richness to have informed discourses with humans in natural language; this is particularly true for systems specialized in visually-dense information, such as dialogue, recommendation, and search engines for clothing. To this end, we train a visual question answering (VQA) system to answer complex natural language questions about apparel in fashion photoshoot images. The key to the successful training of our VQA model is the automatic creation of a visual question-answering dataset with 168 million samples from item attributes of 207 thousand images using diverse templates. The sample generation employs a strategy that considers the difficulty of the question-answer pairs to emphasize challenging concepts. Contrary to the recent trends in using several datasets for pretraining the visual question answering models, we focused on keeping the dataset fixed while training various models from scratch to isolate the improvements from model architecture changes. We see that using the same transformer for encoding the question and decoding the answer, as in language models, achieves maximum accuracy, showing that visual language models (VLMs) make the best visual question answering systems for our dataset. The accuracy of the best model surpasses the human expert level, even when answering human-generated questions that are not confined to the template formats. Our approach for generating a large-scale multimodal domain-specific dataset provides a path for training specialized models capable of communicating in natural language. The training of such domain-expert models, e.g., our fashion VLM model, cannot rely solely on the large-scale general-purpose datasets collected from the web.

FashionViL: Fashion-Focused Vision-and-Language Representation Learning

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

FashionFAE: Fine-grained Attributes Enhanced Fashion Vision-Language Pre-training

FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning

Masked Vision-Language Transformer in Fashion

A Fine-Grained Vision and Language Representation Framework with Graph-Based Fashion Semantic Knowledge

UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation

FashionVQA: A Domain-Specific Visual Question Answering System

Fashionformer: A Simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition.

FashionSAP: Symbols and Attributes Prompt for Fine-grained Fashion Vision-Language Pre-training

Fashion Meets Computer Vision

Kaleido-BERT: Vision-Language Pre-training on Fashion Domain.

MMFashion: An Open-Source Toolbox for Visual Fashion Analysis

MMFL-Net: Multi-scale and Multi-granularity Feature Learning for Cross-domain Fashion Retrieval

FashionKLIP: Enhancing E-Commerce Image-Text Retrieval with Fashion Multi-Modal Conceptual Knowledge Graph

ViT-Lens: Towards Omni-modal Representations

Self-Supervised Vision Transformer for Enhanced Virtual Clothes Try-On

Enhancing Fashion Classification with Vision Transformer (ViT) and Developing Recommendation Fashion Systems Using DINOVA2

FashionLOGO: Prompting Multimodal Large Language Models for Fashion Logo Embeddings

Fashion-VDM: Video Diffusion Model for Virtual Try-On

AABLSTM: A Novel Multi-task Based CNN-RNN Deep Model for Fashion Analysis.