Abstract:In this paper, we propose a novel neural network structure, namely feedforward sequential memory networks (FSMN), to model long-term dependence in time series without using recurrent feedback. The proposed FSMN is a standard fully connected feedforward neural network equipped with some learnable memory blocks in its hidden layers. The memory blocks use a tapped-delay line structure to encode the long context information into a fixed-size representation as short-term memory mechanism which are somehow similar to the time-delay neural networks layers. We have evaluated the FSMNs in several standard benchmark tasks, including speech recognition and language modeling. Experimental results have shown that FSMNs outperform the conventional recurrent neural networks (RNN) while can be learned much more reliably and faster in modeling sequential signals like speech or language. Moreover, we also propose a compact feedforward sequential memory networks (cFSMN) by combining FSMN with low-rank matrix factorization and make a slight modification to the encoding method used in FSMNs in order to further simplify the network architecture. On the speech recognition Switchboard task, the proposed cFSMN structures can reduce the model size by 60% and speed up the learning by more than seven times while the model can still significantly outperform the popular bidirectional LSTMs for both frame-level cross-entropy criterion-based training and MMI-based sequence training.

The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models.

A Fixed-Size Encoding Method for Variable-Length Sequences with its Application to Neural Network Language Models

Learning FOFE Based FNN-LMs with Noise Contrastive Estimation and Part-of-speech Features

Nonrecurrent Neural Structure for Long-Term Dependence.

Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency.

Feedforward Sequential Memory Neural Networks without Recurrent Feedback

Compact Feedforward Sequential Memory Networks for Large Vocabulary Continuous Speech Recognition

F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Feedforward Sequential Memory Networks Based Encoder-Decoder Model for Machine Translation

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models Via MoE-Style Plugin.

Deep-FSMN for Large Vocabulary Continuous Speech Recognition

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Less-forgetting Multi-lingual Fine-tuning

An Optimization Scheme for Segmented-Memory Neural Network

Neurons in Large Language Models: Dead, N-gram, Positional

Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model

Retentive Network: A Successor to Transformer for Large Language Models

MoEfication: Transformer Feed-forward Layers Are Mixtures of Experts

Locality Sensitive Sparse Encoding for Learning World Models Online

Linear Attention via Orthogonal Memory