Abstract:This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts. Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models. After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI.

The Cost of Training NLP Models: A Concise Overview

Costs to Consider in Adopting NLP for Your Business

The economic trade-offs of large language models: A case study

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

Energy and Policy Considerations for Deep Learning in NLP

The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Models

Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models

The rising costs of training frontier AI models

Energy and Policy Considerations for Modern Deep Learning Research

SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees

Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training

On Estimating the Training Cost of Conversational Recommendation Systems

Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations

The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures

Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data

Low-Cost Language Models: Survey and Performance Evaluation on Python Code Generation

Foundation Models for Natural Language Processing -- Pre-trained Language Models Integrating Media

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation