Tuning a Sequence Classification model on the IMDB Dataset
 

Introduction

In this tutorial we will be showing an end-to-end example of fine-tuning a Transformer for sequence classification on a custom dataset in CSV file format.

By the end of this you should be able to:

  1. Build a dataset with the SequenceClassificationDatasets class, and their DataLoaders
  2. Build a SequenceClassificationTuner quickly, find a good learning rate, and train with the One-Cycle Policy
  3. Save that model away, to be used with deployment or other HuggingFace libraries
  4. Apply inference using both the Tuner available function as well as with the EasySequenceClassifier class within AdaptNLP

Installing the Library

This tutorial utilizies the latest AdaptNLP version, as well as parts of the fastai library. Please run the below code to install them:

!pip install adaptnlp -U

(or pip3)

Getting the Dataset

First we need a dataset. We will use the fastai library to download the IMDB_SAMPLE dataset, a subset of IMDB Movie Reviews.

from fastai.data.external import URLs, untar_data

URLs holds a namespace of many data endpoints, and untar_data is a function that can download and extract any data from a given URL.

Combining both, we can download the data:

data_path = untar_data(URLs.IMDB_SAMPLE)

If we look at what was downloaded, we will find a texts.csv file:

data_path.ls()
(#1) [Path('/root/.fastai/data/imdb_sample/texts.csv')]

This is our data we want to use. This CSV is formatted with a table of columns with label, text, and is_valid dictating whether it is part of the validation set or not.

Now that we have the dataset, and we know the format it is in, let's pick a viable model to train with

Picking a Model with the Hub

AdaptNLP has a HFModelHub class that allows you to communicate with the HuggingFace Hub and pick a model from it, as well as a namespace HF_TASKS class with a list of valid tasks we can search by.

Let's try and find one suitable for sequence classification.

First we need to import the class and generate an instance of it:

from adaptnlp import HFModelHub, HF_TASKS
hub = HFModelHub()

Next we can search for a model:

models = hub.search_model_by_task(
    task=HF_TASKS.TEXT_CLASSIFICATION
)

Let's look at a few:

models[:10]
[Model Name: distilbert-base-uncased-finetuned-sst-2-english, Tasks: [text-classification],
 Model Name: roberta-base-openai-detector, Tasks: [text-classification],
 Model Name: roberta-large-mnli, Tasks: [text-classification],
 Model Name: roberta-large-openai-detector, Tasks: [text-classification]]

These are models specifically tagged with the text-classification tag, so you may not see a few models you would expect such as bert_base_cased.

We'll use that first model, distilbert-base-uncased:

model = models[0]
model
Model Name: distilbert-base-uncased-finetuned-sst-2-english, Tasks: [text-classification]

Now that we have picked a model, let's use the data API to prepare our data

Each task has a high-level data wrapper around the TaskDatasets class. In our case this is the SequenceClassificationDatasets class:

from adaptnlp import SequenceClassificationDatasets

There are multiple different constructors for the SequenceClassificationDatasets class, and you should never call the main constructor directly.

We will be using from_csvs, which wraps around the from_dfs constructor:

SequenceClassificationDatasets.from_csvs[source]

SequenceClassificationDatasets.from_csvs(train_csv:Path, text_col:str, label_col:str, tokenizer_name:str, tokenize:bool=True, is_multicategory:bool=False, label_delim=' ', valid_csv:Path=None, split_func=None, split_pct=0.2, tokenize_kwargs:dict={}, auto_kwargs:dict={}, **kwargs)

Builds SequenceClassificationDatasets from a single csv or set of csvs. A convience constructor for from_dfs

Parameters:

  • train_csv : <class 'pathlib.Path'>

    A training csv file

  • text_col : <class 'str'>

    The name of the text column

  • label_col : <class 'str'>

    The name of the label column

  • tokenizer_name : <class 'str'>

    The name of the tokenizer

  • tokenize : <class 'bool'>, optional

    Whether to tokenize immediatly

  • is_multicategory : <class 'bool'>, optional

    Whether each item has a single label or multiple labels

  • label_delim : <class 'str'>, optional

    If `is_multicategory`, how to separate the labels

  • valid_csv : <class 'pathlib.Path'>, optional

    An optional validation csv

  • split_func : <class 'NoneType'>, optional

    Optionally a splitting function similar to RandomSplitter

  • split_pct : <class 'float'>, optional

    What % to split the train_df

  • tokenize_kwargs : <class 'dict'>, optional

    kwargs for the tokenize function

  • auto_kwargs : <class 'dict'>, optional

    kwargs for the AutoTokenizer.from_pretrained constructor

  • kwargs : <class 'inspect._empty'>

Anything you would normally pass to the tokenizer call (such as max_length, padding) should go in tokenize_kwargs, and anything going to the AutoTokenizer.from_pretrained constructor should be passed to the auto_kwargs.

In our case we only have a train_csv, we want to split based on a column, and we have a tokenizer name.

First we'll write a basic function that takes in some items in the form of a later opened DataFrame, and returns two lists of train_idxs, and valid_idxs

import pandas as pd
def get_y(items:pd.DataFrame):
    idxs = (items["is_valid"].values.astype('bool'))
    train_idxs, valid_idxs = [], []
    for i,idx in enumerate(idxs):
        train_idxs.append(i) if idx else valid_idxs.append(i)
    return (train_idxs, valid_idxs)

Next we'll build our SequenceClassificationDatasets:

dsets = SequenceClassificationDatasets.from_csvs(
    data_path/'texts.csv',
    text_col='text',
    label_col='label',
    tokenizer_name=model.name,
    tokenize=True,
    split_func=get_y,
    tokenize_kwargs={'max_length':128, 'truncation':True, 'padding':True}
)

And finally turn it into some AdaptiveDataLoaders.

These are just fastai's DataLoaders class, but it overrides a few functions to have it work nicely with HuggingFace's Dataset class

SequenceClassificationDatasets.dataloaders[source]

SequenceClassificationDatasets.dataloaders(batch_size=8, shuffle_train=True, collate_fn=None, path='.', device=None)

Build DataLoaders from self

Parameters:

  • batch_size : <class 'int'>, optional

    A batch size

  • shuffle_train : <class 'bool'>, optional

    Whether to shuffle the training dataset

  • collate_fn : <class 'NoneType'>, optional

    A custom collation function

  • path : <class 'str'>, optional

  • device : <class 'NoneType'>, optional

dls = dsets.dataloaders(batch_size=8)

Finally, let's view a batch of data with the show_batch function:

dls.show_batch()
Input Label
0 once again the same familiar story about a man ( writer here ) who sell his soul to the devil in order to have his most desired ambition in life : success. unfunny script ( we should " go home and write better " ), ridiculous lines in order to understand the " strong " " christmanish " message ( our only aspiration in life is to find love, respect and a good friendship ) and a very long trial scene at the end where the agent hopkins beat the devil ( jennifer love hewitt is no sexy or evil at all ) for all the bad things she made to this unlikable character. negative
1 i'm not sure under what circumstances director visconti decided to film james cain's novel " the postman always rings twice " ( i'm not even sure if viscounti acquired the book's rights ), but the resulting movie is definitely interesting. it is not the best version of cain's story ( i like the 1981 version best ), but thanks to visconti's excellent direction and the casting of clara calamai and massimo girotti ( a very sensual couple ), it is a must for noir fans. visconti mixes neorealism with noir sensibilities to great effect positive
2 one of the joys of picking up the recent bela lugosi collection is getting to see delightful movies like the invisible ray. boris karloff and bela lugosi team up in a movie that delves into meteorites and radiation and while the science is all perfectly absurd ( especially the camera technique karloff, as janos rukh, uses to determine the site of a certain meteorite ) and downright laughable, i didn't care in the lease because the movie is thoroughly enjoyable. the effects are done well for the time, the acting is great, and the finish is particularly strong. it reminds me of positive
3 i have just lost three hours of my life to this travesty, and i can honestly say i feel violated. i had read the reviews and heard the warnings, and i thought i was prepared for anything - at best i thought, a faithful ( if misguided ) attempt at an original adaptation ; at worst, a so - good - it's - bad " plan 9 " for the new millennium. so when i managed to pick up a copy in walmart while in florida and brought it back to the uk, i joked to my friends " prepare for the worst movie ever made! " oh, cruel karma negative
4 a lot of people seemed to have liked the film, so i feel somewhat bad giving it a bad review. but after sitting through 96 minutes of it, i feel i have to do so. where the heck is the plot in this film?! i must have missed it, i was waiting for the storyline to unfold and nothing happened. sure the ending was " somewhat shocking " but they didn't build up to it. i forgot who was who half of the time, so they didn't really develop the characters. the acting was so - so, most of the time it was believable, but i negative

Building Tuner

Next we need to build a compatible Tuner for our problem. These tuners contain good defaults for our problem space, including loss functions and metrics.

First let's import the SequenceClassificationTuner and view it's documentation

from adaptnlp import SequenceClassificationTuner

class SequenceClassificationTuner[source]

SequenceClassificationTuner(dls:DataLoaders, model_name:str, tokenizer=None, loss_func=CrossEntropyLoss(), metrics=[<function accuracy at 0x7f62125b3820>, <fastai.metrics.AccumMetric object at 0x7f62123e20a0>], opt_func=Adam, additional_cbs=None, expose_fastai_api=False, num_classes:int=None, **kwargs) :: AdaptiveTuner

An AdaptiveTuner with good defaults for Sequence Classification tasks

Valid kwargs and defaults:

  • lr:float = 0.001
  • splitter:function = trainable_params
  • cbs:list = None
  • path:Path = None
  • model_dir:Path = 'models'
  • wd:float = None
  • wd_bn_bias:bool = False
  • train_bn:bool = True
  • moms: tuple(float) = (0.95, 0.85, 0.95)

Parameters:

  • dls : <class 'fastai.data.core.DataLoaders'>

    A set of DataLoaders

  • model_name : <class 'str'>

    A HuggingFace model

  • tokenizer : <class 'NoneType'>, optional

    A HuggingFace tokenizer

  • loss_func : <class 'fastai.losses.CrossEntropyLossFlat'>, optional

    A loss function

  • metrics : <class 'list'>, optional

    Metrics to monitor the training with

  • opt_func : <class 'function'>, optional

    A fastai or torch Optimizer

  • additional_cbs : <class 'NoneType'>, optional

    Additional Callbacks to have always tied to the Tuner,

  • expose_fastai_api : <class 'bool'>, optional

    Whether to expose the fastai API

  • num_classes : <class 'int'>, optional

    The number of classes

  • kwargs : <class 'inspect._empty'>

Next we'll pass in our DataLoaders, the name of our model, and the tokenizer:

tuner = SequenceClassificationTuner(dls, model.name)

By default we can see that it used CrossEntropyLoss as our loss function, and both accuracy and F1Score as our metrics:

tuner.loss_func
FlattenedLoss of CrossEntropyLoss()
_ = [print(m.name) for m in tuner.metrics]
accuracy
f1_score

It is also possible to define your own metrics, these stem from fastai.

To do so, write a function that takes an input and an output, and performs an operation. For example, we will write our own accuracy metric:

def ourAccuracy(inp, out):
    "A simplified accuracy metric that doesn't flatten"
    return (inp == targ).float().mean()

And then we pass it into the constructor:

tuner = SequenceClassificationTuner(dls, model.name, metrics=[ourAccuracy])

If we look at the metrics, you can see that now it is just ourAccuracy:

tuner.metrics[0].name
'ourAccuracy'

For this tutorial, we will revert it back to the defaults:

tuner = SequenceClassificationTuner(dls, model.name)

Finally we just need to train our model!

Fine-Tuning

And all that's left is to tune. There are only 4 or 5 functions you can call on our tuner currently, and this is by design to make it simplistic. In case you don't want to be boxed in however, if you pass in expose_fastai_api=True to our earlier call, it will expose the entirety of Learner to you, so you can call fit_one_cycle, lr_find, and everything else as Tuner uses fastai under the hood.

First, let's call lr_find, which uses fastai's Learning Rate Finder to help us pick a learning rate.

AdaptiveTuner.lr_find[source]

AdaptiveTuner.lr_find(start_lr=1e-07, end_lr=10, num_it=100, stop_div=True, show_plot=True, suggest_funcs=valley)

Runs fastai's LR Finder

Parameters:

  • start_lr : <class 'float'>, optional

  • end_lr : <class 'int'>, optional

  • num_it : <class 'int'>, optional

  • stop_div : <class 'bool'>, optional

  • show_plot : <class 'bool'>, optional

  • suggest_funcs : <class 'function'>, optional

tuner.lr_find()
/opt/venv/lib/python3.8/site-packages/fastai/callback/schedule.py:270: UserWarning: color is redundantly defined by the 'color' keyword argument and the fmt string "ro" (-> color='r'). The keyword argument will take precedence.
  ax.plot(val, idx, 'ro', label=nm, c=color)
SuggestedLRs(valley=0.00015848931798245758)

It recommends a learning rate of around 1e-4, so we will use that.

lr = 1e-4

Let's look at the documentation for tune function:

AdaptiveTuner.tune[source]

AdaptiveTuner.tune(epochs:int, lr:float=None, strategy:Strategy='fit_one_cycle', callbacks:list=[], **kwargs)

Fine tune self.model for epochs with an lr and strategy

Parameters:

  • epochs : <class 'int'>

    Number of iterations to train for

  • lr : <class 'float'>, optional

    If None, finds a new learning rate and uses suggestion_method

  • strategy : <class 'fastcore.basics.Strategy'>, optional

    A fitting method

  • callbacks : <class 'list'>, optional

    Extra fastai Callbacks

  • kwargs : <class 'inspect._empty'>

We can pass in a number of epochs, a learning rate, a strategy, and additional fastai callbacks to call.

Valid strategies live in the Strategy namespace class, and consist of:

from adaptnlp import Strategy

In this tutorial we will train with the One-Cycle policy.

tuner.tune(3, lr, strategy=Strategy.OneCycle)
epoch train_loss valid_loss accuracy f1_score time
0 0.611555 0.435229 0.810000 0.781609 00:05
1 0.367382 0.578555 0.782500 0.730650 00:05
2 0.209047 0.494948 0.825000 0.807692 00:05

Saving Model

Now that we have a trained model, let's save those weights away.

Calling tuner.save will save both the model and the tokenizer in the same format as how HuggingFace does:

AdaptiveTuner.save[source]

AdaptiveTuner.save(save_directory)

Save a pretrained model to a save_directory

Parameters:

  • save_directory : <class 'inspect._empty'>

    A folder to save our model to

tuner.save('good_model')
'good_model'

Performing Inference

There are two ways to get predictions, the first is with the .predict method in our tuner. This is great for if you just finished training and want to see how your model performs on some new data! The other method is with AdaptNLP's inference API, which we will show afterwards

In Tuner

First let's write a sentence ot test with

sentence = "This movie was horrible! Hugh Jackman is a terrible actor"

And then predict with it:

SequenceClassificationTuner.predict[source]

SequenceClassificationTuner.predict(text:Union[List[str], str], bs:int=64, detail_level:DetailLevel='low', class_names:list=None)

Predict some text for sequence classification with the currently loaded model

Parameters:

  • text : typing.Union[typing.List[str], str]

    Some text or list of texts to do inference with

  • bs : <class 'int'>, optional

    A batch size to use for multiple texts

  • detail_level : <class 'fastcore.basics.DetailLevel'>, optional

    A detail level to return on the predictions

  • class_names : <class 'list'>, optional

    A list of labels

Returns:

  • <class 'dict'>

    A dictionary of filtered predictions

tuner.predict(sentence)
{'sentences': ['This movie was horrible! Hugh Jackman is a terrible actor'],
 'predictions': ['negative'],
 'probs': tensor([[0.9935, 0.0065]])}

With the Inference API

Next we will use the EasySequenceClassifier class, which AdaptNLP offers:

from adaptnlp import EasySequenceClassifier

We simply construct the class:

classifier = EasySequenceClassifier()

And call the tag_text method, passing in the sentence, the location of our saved model, and some names for our classes:

classifier.tag_text(
    sentence,
    model_name_or_path='good_model',
    class_names=['negative', 'positive']
)
2021-08-02 14:37:46,609 loading file good_model
{'sentences': ['This movie was horrible! Hugh Jackman is a terrible actor'],
 'predictions': ['negative'],
 'probs': tensor([[0.9935, 0.0065]])}

And we got the exact same output and probabilities!

There are also different levels of predictions we can return (which is also the same with our earlier predict call).

These live in a namespace DetailLevel class, with a few examples below:

from adaptnlp import DetailLevel
DetailLevel.Low
'low'

While some Easy modules will not return different items at each level, most will return only a few specific outputs at the Low level, and everything possible at the High level:

classifier.tag_text(
    sentence,
    model_name_or_path = 'good_model',
    detail_level=DetailLevel.Low
)
{'sentences': ['This movie was horrible! Hugh Jackman is a terrible actor'],
 'predictions': ['NEGATIVE'],
 'probs': tensor([[0.9935, 0.0065]])}
classifier.tag_text(
    sentence,
    model_name_or_path = 'good_model',
    detail_level=DetailLevel.Medium
)
{'sentences': ['This movie was horrible! Hugh Jackman is a terrible actor'],
 'predictions': ['NEGATIVE'],
 'probs': tensor([[0.9935, 0.0065]]),
 'pairings': OrderedDict([('This movie was horrible! Hugh Jackman is a terrible actor',
               tensor([0.9935, 0.0065]))]),
 'classes': ['NEGATIVE', 'POSITIVE']}
classifier.tag_text(
    sentence,
    model_name_or_path = 'good_model',
    detail_level=DetailLevel.High
)
{'sentences': [Sentence: "This movie was horrible ! Hugh Jackman is a terrible actor"   [− Tokens: 11  − Sentence-Labels: {'sc': [NEGATIVE (0.9935), POSITIVE (0.0065)]}]],
 'predictions': ['NEGATIVE'],
 'probs': tensor([[0.9935, 0.0065]]),
 'pairings': OrderedDict([('This movie was horrible! Hugh Jackman is a terrible actor',
               tensor([0.9935, 0.0065]))]),
 'classes': ['NEGATIVE', 'POSITIVE']}