Abstract:Causal decoder-only transformer models used for generative language modelling, such as Generative Pre-trained Transformers (GPT), are trained to predict the next token in a sequence based only on its previous tokens. Despite this simple training objective, they have proved to be powerful AI tools. However, only predicting the next token results in top layer embedding vectors that are highly token-focused. There may be benefits in generating embedding vectors at each token position that better capture the overall meaning of longer sequences of future text. Recent studies matching brain scans with deep language models suggest that humans also predict upcoming words when listening or reading but consider multiple future tokens rather than just one. This research investigates a new pretraining method called Future Token Prediction (FTP). In FTP, a large transformer encoder generates top layer embedding vectors for each token position, which, instead of being passed to a language head, are linearly and expansively projected to a pseudo-sequence, which is cross attended to by a small transformer decoder to predict the next N tokens forward from that position in the sequence. The top layer embedding vectors from FTP models exhibit distinct properties compared to those from standard GPT models, varying smoothly along a text sequence as measured by cosine similarity between adjacent tokens. Text generated by FTP models show improved topic coherence compared to standard GPT-like models trained with the same prediction perplexity for the next single token. The vectors are shown to better represent the topic of text based on the results of text classification examples. On a toy, but complex, coding problem, FTP networks produce significantly better results than GPT networks.

Predicting Next Application Most Likely Used with Word Embedding and Time-Series Data Encoding

Predictive Analysis by Leveraging Temporal User Behavior and User Embeddings

Next Word Prediction Using Deep Learning: A Comparative Study

A Novel Deep Learning-Based Encoder-Decoder Model for Remaining Useful Life Prediction

Predicting and Recommending the Next Smartphone Apps Based on Recurrent Neural Network

Time-Dependent Representation for Neural Event Sequence Prediction

ATPP: A Mobile App Prediction System Based on Deep Marked Temporal Point Processes

Gene2vec: gene subsequence embedding for prediction of mammalian N 6 -methyladenosine sites from mRNA.

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction

NGCU: A New RNN Model for Time-Series Data Prediction

Efficient Sequence Transduction by Jointly Predicting Tokens and Durations

App2Vec: Context-Aware Application Usage Prediction.

DeepAPP: A Deep Reinforcement Learning Framework for Mobile Application Usage Prediction

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Time-Series Embeddings from Language Models: A Tool for Wind Direction Nowcasting

Learning Abstract Snippet Detectors with Temporal Embedding in Convolutional Neural Networks

What to Do Next: Modeling User Behaviors by Time-LSTM

TEALED: A Multi-Step Workload Forecasting Approach Using Time-Sensitive EMD and Auto LSTM Encoder-Decoder

AppUsage2Vec: Modeling Smartphone App Usage for Prediction.

DeepApp

Machine Learning of Time Series Using Time-Delay Embedding and Precision Annealing