Abstract:Bidirectional attention $\unicode{x2013}$ composed of self-attention with positional encodings and the masked language model (MLM) objective $\unicode{x2013}$ has emerged as a key component of modern large language models (LLMs). Despite its empirical success, few studies have examined its statistical underpinnings: What statistical model is bidirectional attention implicitly fitting? What sets it apart from its non-attention predecessors? We explore these questions in this paper. The key observation is that fitting a single-layer single-head bidirectional attention, upon reparameterization, is equivalent to fitting a continuous bag of words (CBOW) model with mixture-of-experts (MoE) weights. Further, bidirectional attention with multiple heads and multiple layers is equivalent to stacked MoEs and a mixture of MoEs, respectively. This statistical viewpoint reveals the distinct use of MoE in bidirectional attention, which aligns with its practical effectiveness in handling heterogeneous data. It also suggests an immediate extension to categorical tabular data, if we view each word location in a sentence as a tabular feature. Across empirical studies, we find that this extension outperforms existing tabular extensions of transformers in out-of-distribution (OOD) generalization. Finally, this statistical perspective of bidirectional attention enables us to theoretically characterize when linear word analogies are present in its word embeddings. These analyses show that bidirectional attention can require much stronger assumptions to exhibit linear word analogies than its non-attention predecessors.

Bi-Directional Recurrent Attentional Topic Model

Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data

Neural Topic Modeling with Bidirectional Adversarial Training

Recurrent Attention Networks for Long-text Modeling

Utilizing Recurrent Neural Network for Topic Discovery in Short Text Scenarios

Deep Autoencoding Topic Model With Scalable Hybrid Bayesian Inference

A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

Bidirectional Attentional Encoder-Decoder Model and Bidirectional Beam Search for Abstractive Summarization

Topic Model over Short Texts Incorporating Word Embedding

A Joint Model Of Extended Lda And Ibtm Over Streaming Chinese Short Texts

Recurrent Neural Network Language Model Adaptation Derived Document Vector

TDAM: a Topic-Dependent Attention Model for Sentiment Analysis

Tag-Weighted Topic Model for Mining Semi-Structured Documents.

A Topic Model for Linked Documents and Update Rules for Its Estimation

TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Joint sentiment/topic modeling on text data using a boosted restricted Boltzmann Machine

An Attention Hierarchical Topic Modeling

Bidirectional Attention as a Mixture of Continuous Word Experts

Neural Topic Modeling with Cycle-Consistent Adversarial Training

Multi-Document Summarization Using Sentence-based Topic Models.

Relational Biterm Topic Model: Short-Text Topic Modeling Using Word Embeddings