Abstract:This thesis provides methods and analysis of models which make progress on this goal. The techniques outlined are task agnostic, and should provide benefit when used with nearly any transformer LM. We introduce two new finetuning methods which add new capabilities to the models they are used on. The first adds a recurrence mechanism, which removes the fixed-window sized constraint and improves the efficiency of a transformer decoder. The second allows masked language models (MLMs) to be used for initialization of both the encoder and decoder of a non-autoregressive sequence-to-sequence transformer, opening up generative applications of models which were previously only used for natural language understanding tasks. We also introduce two new techniques for improving the quality of predictions of any transformer decoder without additional finetuning. One, hidden state optimization, can be applied to any transformer decoder to improve the quality of predictions at inference time, especially for few-shot classification. The other, conditional beam search, allows practitioners to search for natural language generation (NLG) model outputs with high likelihood while conditioning on the event that the output is not degenerate (e.g. empty, repetitive, etc.). Finally, we provide theoretical and empirical insights on the divergence of model-likelihood and output quality which has widely been observed in prior work. These insights apply to any model which represents a distribution over text, and apply to language models which are not transformers or even autoregressive. We argue that the NLP community has, to some extent, misunderstood the implications of these findings, and encourage a point of view which has more nuance.

Rethinking the adaptive relationship between Encoder Layers and Decoder Layers

Learning to Decode for Future Success

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translation

Rethinking Skip Connection with Layer Normalization in Transformers and ResNets

Length bias in Encoder Decoder Models and a Case for Global Conditioning

Is Encoder-Decoder Redundant for Neural Machine Translation?

Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training

Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers

Balancing Cost and Benefit with Tied-Multi Transformers

Trimming and Improving Skip-thought Vectors

Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages

Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

Layer-wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-trained Models: An Evolutionary Approach

Rethinking Residual Connection with Layer Normalization

Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation

Dynamic layer selection in decoder-only transformers

Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers

One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization