Abstract:Models of word meaning that exploit patterns of word usage across large text corpora to capture semantic relations, like the topic model and word2vec, condense word-by-context co-occurrence statistics to induce representations that organize words along semantically relevant dimensions (e.g., synonymy, antonymy, hyponymy, etc.). However, their reliance on latent representations leaves them vulnerable to interference, makes them slow learners, and commits to a dual-systems account of episodic and semantic memory. We show how it is possible to construct the meaning of words online during retrieval to avoid these limitations. We implement a spreading activation account of word meaning in an associative net, a one-layer highly recurrent network of associations, called a Dynamic-Eigen-Net, that we developed to address the limitations of earlier variants of associative nets when scaling up to deal with unstructured input domains like natural language text. We show that spreading activation using a one-hot coded Dynamic-Eigen-Net outperforms the topic model and reaches similar levels of performance as word2vec when predicting human free associations and word similarity ratings. Latent Semantic Analysis vectors reached similar levels of performance when constructed by applying dimensionality reduction to the Shifted Positive Pointwise Mutual Information but showed poorer predictability for free associations when using an entropy-based normalization. An analysis of the rate at which the Dynamic-Eigen-Net reaches asymptotic performance shows that it learns faster than word2vec. We argue in favor of the Dynamic-Eigen-Net as a fast learner, with a single-store, that is not subject to catastrophic interference. We present it as an alternative to instance models when delegating the induction of latent relationships to process assumptions instead of assumptions about representation.

Deep Word Association: A Flexible Chinese Word Association Method with Iterative Attention Mechanism.

Deep Latent Variable Predictive Modeling with Online Bayesian Soft Attention Mechanism

Associated words recognition in Chinese compound sentences based on deep learning

SAC: Accelerating and Structuring Self-Attention Via Sparse Adaptive Connection.

Neural Machine Translation with Deep Attention

An Adaptive Wordpiece Language Model For Learning Chinese Word Embeddings

SCCA: Shifted Cross Chunk Attention for long contextual semantic expansion

A Context-focused Attention Evolution Model for Aspect-based Sentiment Classification

A Long Dependency Aware Deep Architecture for Joint Chinese Word Segmentation and POS Tagging.

Constructing Word Association Network by Crowdsourcing

Named Entity Recognition In Chinese Electronic Medical Record Using Attention Mechanism

Combination Methods of Chinese Character and Word Embeddings in Deep Learning

Optimizing Automatic Text Classification Approach in Adaptive Online Collaborative Discussion–A Perspective of Attention Mechanism-Based Bi-LSTM

Learning to Associate Words and Images Using a Large-scale Graph

Word Attention for Sequence to Sequence Text Understanding.

Towards Better Text-to-Image Generation Alignment via Attention Modulation

Deep Learning for Chinese Word Segmentation and POS Tagging.

An Iterative Associative Memory Model for Empathetic Response Generation

DeepFocus: Deep Encoding Brainwaves and Emotions with Multi-Scenario Behavior Analytics for Human Attention Enhancement

Latent Relations at Steady-state with Associative Nets

Leveraging Gloss Knowledge in Neural Word Sense Disambiguation by Hierarchical Co-Attention