Summary of the tasks

This page shows the most frequent use-cases when using the library. The models available allow for many different configurations and a great versatility in use-cases. The most simple ones are presented here, showcasing usage for tasks such as image classification, question answering, sequence classification, named entity recognition and others.

These examples leverage auto-models, which are classes that will instantiate a model according to a given checkpoint, automatically selecting the correct model architecture. Please check the AutoModel documentation for more information. Feel free to modify the code to be more specific and adapt it to your specific use-case.

In order for a model to perform well on a task, it must be loaded from a checkpoint corresponding to that task. These checkpoints are usually pre-trained on a large corpus of data and fine-tuned on a specific task. This means the following:

• Not all models were fine-tuned on all tasks. If you want to fine-tune a model on a specific task, you can leverage one of the run_$TASK.py scripts in the examples directory.
• Fine-tuned models were fine-tuned on a specific dataset. This dataset may or may not overlap with your use-case and domain. As mentioned previously, you may leverage the examples scripts to fine-tune your model, or you may create your own training script.

In order to do an inference on a task, several mechanisms are made available by the library:

• Pipelines: very easy-to-use abstractions, which require as little as two lines of code.
• Direct model use: Less abstractions, but more flexibility and power via a direct access to a tokenizer (PyTorch/TensorFlow) and full inference capacity.

Both approaches are showcased here.

Sequence Classification

Sequence classification is the task of classifying sequences according to a given number of classes. An example of sequence classification is the GLUE dataset, which is entirely based on that task. If you would like to fine-tune a model on a GLUE sequence classification task, you may leverage the run_glue.py, run_tf_glue.py, run_tf_text_classification.py or run_xnli.py scripts.

Here is an example of using pipelines to do sentiment analysis: identifying if a sequence is positive or negative. It leverages a fine-tuned model on sst2, which is a GLUE task.

This returns a label ("POSITIVE" or "NEGATIVE") alongside a score, as follows:

Here is an example of doing a sequence classification using a model to determine if two sequences are paraphrases of each other. The process is the following:

1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it with the weights stored in the checkpoint.
2. Build a sequence from the two sentences, with the correct model-specific separators, token type ids and attention masks (which will be created automatically by the tokenizer).
3. Pass this sequence through the model so that it is classified in one of the two available classes: 0 (not a paraphrase) and 1 (is a paraphrase).
4. Compute the softmax of the result to get probabilities over the classes.
5. Print the results.

Extractive Question Answering

Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune a model on a SQuAD task, you may leverage the run_qa.py and run_tf_squad.py scripts.

Here is an example of using pipelines to do question answering: extracting an answer from a text given a question. It leverages a fine-tuned model on SQuAD.

This returns an answer extracted from the text, a confidence score, alongside "start" and "end" values, which are the positions of the extracted answer in the text.

Here is an example of question answering using a model and a tokenizer. The process is the following:

1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it with the weights stored in the checkpoint.
2. Define a text and a few questions.
3. Iterate over the questions and build a sequence from the text and the current question, with the correct model-specific separators, token type ids and attention masks.
4. Pass this sequence through the model. This outputs a range of scores across the entire sequence tokens (question and text), for both the start and end positions.
5. Compute the softmax of the result to get probabilities over the tokens.
6. Fetch the tokens from the identified start and stop values, convert those tokens to a string.
7. Print the results.

Language Modeling

Language modeling is the task of fitting a model to a corpus, which can be domain specific. All popular transformer-based models are trained using a variant of language modeling, e.g. BERT with masked language modeling, GPT-2 with causal language modeling.

Language modeling can be useful outside of pretraining as well, for example to shift the model distribution to be domain-specific: using a language model trained over a very large corpus, and then fine-tuning it to a news dataset or on scientific papers e.g. LysandreJik/arxiv-nlp.

Masked Language Modeling

Masked language modeling is the task of masking tokens in a sequence with a masking token, and prompting the model to fill that mask with an appropriate token. This allows the model to attend to both the right context (tokens on the right of the mask) and the left context (tokens on the left of the mask). Such a training creates a strong basis for downstream tasks requiring bi-directional context, such as SQuAD (question answering, see Lewis, Lui, Goyal et al., part 4.2). If you would like to fine-tune a model on a masked language modeling task, you may leverage the run_mlm.py script.

Here is an example of using pipelines to replace a mask from a sequence:

This outputs the sequences with the mask filled, the confidence score, and the token id in the tokenizer vocabulary:

Here is an example of doing masked language modeling using a model and a tokenizer. The process is the following:

1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a DistilBERT model and loads it with the weights stored in the checkpoint.
2. Define a sequence with a masked token, placing the tokenizer.mask_token instead of a word.
3. Encode that sequence into a list of IDs and find the position of the masked token in that list.
4. Retrieve the predictions at the index of the mask token: this tensor has the same size as the vocabulary, and the values are the scores attributed to each token. The model gives higher score to tokens it deems probable in that context.
5. Retrieve the top 5 tokens using the PyTorch topk or TensorFlow top_k methods.
6. Replace the mask token by the tokens and print the results

This prints five sequences, with the top 5 tokens predicted by the model.

Causal Language Modeling

Causal language modeling is the task of predicting the token following a sequence of tokens. In this situation, the model only attends to the left context (tokens on the left of the mask). Such a training is particularly interesting for generation tasks. If you would like to fine-tune a model on a causal language modeling task, you may leverage the run_clm.py script.

Usually, the next token is predicted by sampling from the logits of the last hidden state the model produces from the input sequence.

Here is an example of using the tokenizer and model and leveraging the top_k_top_p_filtering() method to sample the next token following an input sequence of tokens.

Here is an example of using the tokenizer and model and leveraging the tf_top_k_top_p_filtering() method to sample the next token following an input sequence of tokens.

This outputs a (hopefully) coherent next token following the original sequence, which in our case is the word is or features.

In the next section, we show how generation.GenerationMixin.generate() can be used to generate multiple tokens up to a specified length instead of one token at a time.

Text Generation

In text generation (a.k.a open-ended text generation) the goal is to create a coherent portion of text that is a continuation from the given context. The following example shows how GPT-2 can be used in pipelines to generate text. As a default all models apply Top-K sampling when used in pipelines, as configured in their respective configurations (see gpt-2 config for example).

Here, the model generates a random text with a total maximal length of 50 tokens from context "As far as I am concerned, I will". Behind the scenes, the pipeline object calls the method PreTrainedModel.generate() to generate text. The default arguments for this method can be overridden in the pipeline, as is shown above for the arguments max_length and do_sample.

Below is an example of text generation using XLNet and its tokenizer, which includes calling generate() directly:

Text generation is currently possible with GPT-2, OpenAi-GPT, CTRL, XLNet, Transfo-XL and Reformer in PyTorch and for most models in Tensorflow as well. As can be seen in the example above XLNet and Transfo-XL often need to be padded to work well. GPT-2 is usually a good choice for open-ended text generation because it was trained on millions of webpages with a causal language modeling objective.

For more information on how to apply different decoding strategies for text generation, please also refer to our text generation blog post here.

Named Entity Recognition

Named Entity Recognition (NER) is the task of classifying tokens according to a class, for example, identifying a token as a person, an organisation or a location. An example of a named entity recognition dataset is the CoNLL-2003 dataset, which is entirely based on that task. If you would like to fine-tune a model on an NER task, you may leverage the run_ner.py script.

Here is an example of using pipelines to do named entity recognition, specifically, trying to identify tokens as belonging to one of 9 classes:

• O, Outside of a named entity
• B-MIS, Beginning of a miscellaneous entity right after another miscellaneous entity
• I-MIS, Miscellaneous entity
• B-PER, Beginning of a person's name right after another person's name
• I-PER, Person's name
• B-ORG, Beginning of an organisation right after another organisation
• I-ORG, Organisation
• B-LOC, Beginning of a location right after another location
• I-LOC, Location

It leverages a fine-tuned model on CoNLL-2003, fine-tuned by @stefan-it from dbmdz.

This outputs a list of all words that have been identified as one of the entities from the 9 classes defined above. Here are the expected results:

Note how the tokens of the sequence "Hugging Face" have been identified as an organisation, and "New York City", "DUMBO" and "Manhattan Bridge" have been identified as locations.

Here is an example of doing named entity recognition, using a model and a tokenizer. The process is the following:

1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it with the weights stored in the checkpoint.
2. Define a sequence with known entities, such as "Hugging Face" as an organisation and "New York City" as a location.
3. Split words into tokens so that they can be mapped to predictions. We use a small hack by, first, completely encoding and decoding the sequence, so that we're left with a string that contains the special tokens.
4. Encode that sequence into IDs (special tokens are added automatically).
5. Retrieve the predictions by passing the input to the model and getting the first output. This results in a distribution over the 9 possible classes for each token. We take the argmax to retrieve the most likely class for each token.
6. Zip together each token with its prediction and print it.

This outputs a list of each token mapped to its corresponding prediction. Differently from the pipeline, here every token has a prediction as we didn't remove the "0"th class, which means that no particular entity was found on that token.

In the above example, predictions is an integer that corresponds to the predicted class. We can use the model.config.id2label property in order to recover the class name corresponding to the class number, which is illustrated below:

Summarization

Summarization is the task of summarizing a document or an article into a shorter text. If you would like to fine-tune a model on a summarization task, you may leverage the run_summarization.py script.

An example of a summarization dataset is the CNN / Daily Mail dataset, which consists of long news articles and was created for the task of summarization. If you would like to fine-tune a model on a summarization task, various approaches are described in this document.

Here is an example of using the pipelines to do summarization. It leverages a Bart model that was fine-tuned on the CNN / Daily Mail data set.

Because the summarization pipeline depends on the PreTrainedModel.generate() method, we can override the default arguments of PreTrainedModel.generate() directly in the pipeline for max_length and min_length as shown below. This outputs the following summary:

Here is an example of doing summarization using a model and a tokenizer. The process is the following:

1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder model, such as Bart or T5.
2. Define the article that should be summarized.
3. Add the T5 specific prefix "summarize: ".
4. Use the PreTrainedModel.generate() method to generate the summary.

In this example we use Google's T5 model. Even though it was pre-trained only on a multi-task mixed dataset (including CNN / Daily Mail), it yields very good results.

Translation

Translation is the task of translating a text from one language to another. If you would like to fine-tune a model on a translation task, you may leverage the run_translation.py script.

An example of a translation dataset is the WMT English to German dataset, which has sentences in English as the input data and the corresponding sentences in German as the target data. If you would like to fine-tune a model on a translation task, various approaches are described in this document.

Here is an example of using the pipelines to do translation. It leverages a T5 model that was only pre-trained on a multi-task mixture dataset (including WMT), yet, yielding impressive translation results.

Because the translation pipeline depends on the PreTrainedModel.generate() method, we can override the default arguments of PreTrainedModel.generate() directly in the pipeline as is shown for max_length above.

Here is an example of doing translation using a model and a tokenizer. The process is the following:

1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder model, such as Bart or T5.
2. Define the article that should be summarized.
3. Add the T5 specific prefix "translate English to German: "
4. Use the PreTrainedModel.generate() method to perform the translation.

We get the same translation as with the pipeline example.

Audio classification

Audio classification assigns a class to an audio signal. The Keyword Spotting dataset from the SUPERB benchmark is an example dataset that can be used for audio classification fine-tuning. This dataset contains ten classes of keywords for classification. If you'd like to fine-tune a model for audio classification, take a look at the run_audio_classification.py script or this how-to guide.

The following examples demonstrate how to use a pipeline() and a model and tokenizer for audio classification inference:

The general process for using a model and feature extractor for audio classification is:

1. Instantiate a feature extractor and a model from the checkpoint name.
2. Process the audio signal to be classified with a feature extractor.
3. Pass the input through the model and take the argmax to retrieve the most likely class.
4. Convert the class id to a class name with id2label to return an interpretable result.

Automatic speech recognition

Automatic speech recognition transcribes an audio signal to text. The Common Voice dataset is an example dataset that can be used for automatic speech recognition fine-tuning. It contains an audio file of a speaker and the corresponding sentence. If you'd like to fine-tune a model for automatic speech recognition, take a look at the run_speech_recognition_ctc.py or run_speech_recognition_seq2seq.py scripts or this how-to guide.

The following examples demonstrate how to use a pipeline() and a model and tokenizer for automatic speech recognition inference:

The general process for using a model and processor for automatic speech recognition is:

1. Instantiate a processor (which regroups a feature extractor for input processing and a tokenizer for decoding) and a model from the checkpoint name.
2. Process the audio signal and text with a processor.
3. Pass the input through the model and take the argmax to retrieve the predicted text.
4. Decode the text with a tokenizer to obtain the transcription.

Image classification

Like text and audio classification, image classification assigns a class to an image. The CIFAR-100 dataset is an example dataset that can be used for image classification fine-tuning. It contains an image and the corresponding class. If you'd like to fine-tune a model for image classification, take a look at the run_image_classification.py script or this how-to guide.

The following examples demonstrate how to use a pipeline() and a model and tokenizer for image classification inference:

The general process for using a model and image processor for image classification is:

1. Instantiate an image processor and a model from the checkpoint name.
2. Process the image to be classified with an image processor.
3. Pass the input through the model and take the argmax to retrieve the predicted class.
4. Convert the class id to a class name with id2label to return an interpretable result.