Introduction
In this tutorial we will be showing an end-to-end example of fine-tuning a Transformer for sequence classification on a custom dataset in text files format.
By the end of this you should be able to:
- Build a dataset with the
SequenceClassificationDatasets
class, and their DataLoaders - Build a
SequenceClassificationTuner
quickly, find a good learning rate, and train with the One-Cycle Policy - Save that model away, to be used with deployment or other HuggingFace libraries
- Apply inference using both the
Tuner
available function as well as with theEasySequenceClassifier
class within AdaptNLP
This tutorial utilizies the latest AdaptNLP version, as well as parts of the fastai
library. Please run the below code to install them:
!pip install adaptnlp -U
(or pip3
)
First we need a dataset. We will use the fastai
library to download the full IMDB
Movie Reviews dataset
from fastai.data.external import URLs, untar_data
URLs
holds a namespace of many data endpoints, and untar_data
is a function that can download and extract any data from a given URL.
Combining both, we can download the data:
data_path = untar_data(URLs.IMDB)
If we look at what was downloaded, we will find a train
and test
folder:
data_path.ls()
In each are folders seperating each text file by class:
(data_path/'train').ls()
As a result, we can say the dataset follows the following format:
train
class_a
text1.txt
text2.txt
- ...
class_b
text1.txt
- ...
test
(orvalid
)class_a
text1.txt
- ...
class_b
text1.txt
- ...
Now that we have the dataset, and we know the format it is in, let's pick a viable model to train with
Picking a Model with the Hub
AdaptNLP has a HFModelHub
class that allows you to communicate with the HuggingFace Hub and pick a model from it, as well as a namespace HF_TASKS
class with a list of valid tasks we can search by.
Let's try and find one suitable for sequence classification.
First we need to import the class and generate an instance of it:
from adaptnlp import HFModelHub, HF_TASKS
hub = HFModelHub()
Next we can search for a model:
models = hub.search_model_by_task(
task=HF_TASKS.TEXT_CLASSIFICATION
)
Let's look at a few:
models[:10]
These are models specifically tagged with the text-classification
tag, so you may not see a few models you would expect such as bert_base_cased
.
We'll use that first model, distilbert-base-uncased
:
model = models[0]
model
Now that we have picked a model, let's use the data API to prepare our data
Building TaskDatasets
with SequenceClassificationDatasets
Each task has a high-level data wrapper around the TaskDatasets
class. In our case this is the SequenceClassificationDatasets
class:
from adaptnlp import SequenceClassificationDatasets
There are multiple different constructors for the SequenceClassificationDatasets
class, and you should never call the main constructor directly.
We will be using from_folders
method:
Anything you would normally pass to the tokenizer call (such as max_length
, padding
) should go in tokenize_kwargs
, and anything going to the AutoTokenizer.from_pretrained
constructor should be passed to the auto_kwargs
.
In our case we have a train_path
and valid_path
, and the last thing we need to do is write a way to get the label from an individual file.
Let's look at what one of these look like:
item = (data_path/'train'/'pos').ls()[0]
item
So the label is located in the name of the parent relative to the actual file:
item.parent.name
Let's write a quick function to extract that:
.parent.name
functionalityfrom pathlib import Path
def get_y(item:str): return Path(item).parent.name
Next we'll build our SequenceClassificationDatasets
:
dsets = SequenceClassificationDatasets.from_folders(
data_path/'train',
get_label=get_y,
valid_path=data_path/'test',
tokenizer_name=model.name,
tokenize=True,
split_func=get_y,
tokenize_kwargs={'max_length':128, 'truncation':True, 'padding':True}
)
split_func
or split_pct
to either have it split the dataset in a custom way, or pass in a percentage to randomly split byAnd finally turn it into some AdaptiveDataLoaders
.
These are just fastai's DataLoaders
class, but it overrides a few functions to have it work nicely with HuggingFace's Dataset
class
dls = dsets.dataloaders(batch_size=8)
Finally, let's view a batch of data with the show_batch
function:
dls.show_batch()
Next we need to build a compatible Tuner
for our problem. These tuners contain good defaults for our problem space, including loss functions and metrics.
First let's import the SequenceClassificationTuner
and view it's documentation
from adaptnlp import SequenceClassificationTuner
Next we'll pass in our DataLoaders
and the name of our model:
TaskDatasets
, SequenceClassificationDatasets
, etc), you need to pass in the tokenizer to the constructor as well with tokenizer=tokenizer
tuner = SequenceClassificationTuner(dls, model.name)
By default we can see that it used CrossEntropyLoss
as our loss function, and both accuracy
and F1Score
as our metrics:
tuner.loss_func
_ = [print(m.name) for m in tuner.metrics]
It is also possible to define your own metrics, these stem from fastai.
To do so, write a function that takes an input and an output, and performs an operation. For example, we will write our own accuracy
metric:
def ourAccuracy(inp, out):
"A simplified accuracy metric that doesn't flatten"
return (inp == targ).float().mean()
And then we pass it into the constructor:
tuner = SequenceClassificationTuner(dls, model.name, metrics=[ourAccuracy])
If we look at the metrics, you can see that now it is just ourAccuracy
:
tuner.metrics[0].name
For this tutorial, we will revert it back to the defaults:
tuner = SequenceClassificationTuner(dls, model.name)
Finally we just need to train our model!
And all that's left is to tune
. There are only 4 or 5 functions you can call on our tuner
currently, and this is by design to make it simplistic. In case you don't want to be boxed in however, if you pass in expose_fastai_api=True
to our earlier call, it will expose the entirety of Learner
to you, so you can call fit_one_cycle
, lr_find
, and everything else as Tuner
uses fastai
under the hood.
First, let's call lr_find
, which uses fastai's Learning Rate Finder to help us pick a learning rate.
tuner.lr_find()
It recommends a learning rate of around 1e-4, so we will use that.
lr = 1e-4
Let's look at the documentation for tune
function:
We can pass in a number of epochs, a learning rate, a strategy, and additional fastai callbacks to call.
Valid strategies live in the Strategy
namespace class, and consist of:
- OneCycle (Also called the One-Cycle Policy)
- CosineAnnealing
- SGDR
from adaptnlp import Strategy
In this tutorial we will train with the One-Cycle policy, as currently it is one of the best schedulers to use.
Let's now tune
with our strategy and our newly found learning rate for three iterations over the dataset
tuner.tune(
epochs=3,
lr=lr,
strategy=Strategy.OneCycle
)
Now that we have a trained model, let's save those weights away.
Calling tuner.save
will save both the model and the tokenizer in the same format as how HuggingFace does:
tuner.save('good_model')
There are two ways to get predictions, the first is with the .predict
method in our tuner
. This is great for if you just finished training and want to see how your model performs on some new data!
The other method is with AdaptNLP's inference API, which we will show afterwards
First let's write a sentence ot test with
sentence = "This movie was horrible! Hugh Jackman is a terrible actor"
And then predict with it:
tuner.predict(sentence)
Next we will use the EasySequenceClassifier
class, which AdaptNLP offers:
from adaptnlp import EasySequenceClassifier
We simply construct the class:
classifier = EasySequenceClassifier()
And call the tag_text
method, passing in the sentence, the location of our saved model, and some names for our classes:
classifier.tag_text(
sentence,
model_name_or_path='good_model',
class_names=['negative', 'positive']
)
And we got the exact same output and probabilities!
There are also different levels of predictions we can return (which is also the same with our earlier predict
call).
These live in a namespace DetailLevel
class, with a few examples below:
from adaptnlp import DetailLevel
DetailLevel.Low
While some Easy modules will not return different items at each level, most will return only a few specific outputs at the Low level, and everything possible at the High level:
classifier.tag_text(
sentence,
model_name_or_path = 'good_model',
detail_level=DetailLevel.Low
)
classifier.tag_text(
sentence,
model_name_or_path = 'good_model',
detail_level=DetailLevel.Medium
)
classifier.tag_text(
sentence,
model_name_or_path = 'good_model',
detail_level=DetailLevel.High
)