Abstract:In this work, we formulate a visual dialog as an information flow in which each piece of information is encoded with the joint visual-linguistic representation of a single dialog round. Based on this formulation, we consider the visual dialog task as a sequence problem consisting of ordered visual-linguistic vectors. For featurization, we use a Dense Symmetric Co-Attention network as a lightweight vison-language joint representation generator to fuse multimodal features (i.e., image and text), yielding better computation and data efficiencies. For inference, we propose two Sequential Dialog Networks (SeqDialN): the first uses LSTM for information propagation (IP) and the second uses a modified Transformer for multi-step reasoning (MR). Our architecture separates the complexity of multimodal feature fusion from that of inference, which allows simpler design of the inference engine. IP based SeqDialN is our baseline with a simple 2-layer LSTM design that achieves decent performance. MR based SeqDialN, on the other hand, recurrently refines the semantic question/history representations through the self-attention stack of Transformer and produces promising results on the visual dialog task. On VisDial v1.0 test-std dataset, our best single generative SeqDialN achieves 62.54% NDCG and 48.63% MRR; our ensemble generative SeqDialN achieves 63.78% NDCG and 49.98% MRR, which set a new state-of-the-art generative visual dialog model. We fine-tune discriminative SeqDialN with dense annotations and boost the performance up to 72.41% NDCG and 55.11% MRR. In this work, we discuss the extensive experiments we have conducted to demonstrate the effectiveness of our model components. We also provide visualization for the reasoning process from the relevant conversation rounds and discuss our fine-tuning methods. Our code is available at https://github.com/xiaoxiaoheimei/SeqDialN

Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures.

Sequence-to-Sequence Learning for Task-oriented Dialogue with Dialogue State Representation.

Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization

Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL.

A Corpus-Free State2Seq User Simulator for Task-Oriented Dialogue.

Seq2seq is All You Need for Coreference Resolution

End-to-End Task-Oriented Dialogue System with Distantly Supervised Knowledge Base Retriever

Entity-Consistent End-to-end Task-Oriented Dialogue System with KB Retriever

Let Topic Flow: A Unified Topic-Guided Segment-Wise Dialogue Summarization Framework

Fg2seq: Effectively Encoding Knowledge for End-To-End Task-Oriented Dialog.

Tailored Sequence to Sequence Models to Different Conversation Scenarios

Reinforcing Coherence for Sequence to Sequence Model in Dialogue Generation

RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL

Conversational Semantic Parsing

Fast and Scalable Dialogue State Tracking with Explicit Modular Decomposition

Using Textual Interface to Align External Knowledge for End-to-End Task-Oriented Dialogue Systems

A Unified Sequence Interface for Vision Tasks

Modeling Complex Dialogue Mappings Via Sentence Semantic Segmentation Guided Conditional Variational Auto-Encoder

SeqDialN: Sequential Visual Dialog Networks in Joint Visual-Linguistic Representation Space

Memory-to-Sequence learning with LSTM joint decoding for task-oriented dialogue systems

Novel framework for dialogue summarization based on factual-statement fusion and dialogue segmentation