AdaptiveModels for using Transformers and Flair for token tagging and classification

class TokenClassificationResult[source]

TokenClassificationResult(inputs:str, tokenized_inputs:tensor, tagged_entities:dict)

A result class for Token Tagging tasks

Parameters:

  • inputs : <class 'str'>

    Original text inputs

  • tokenized_inputs : <built-in method tensor of type object at 0x7f8ab4249ec0>

    Tokenized inputs

  • tagged_entities : <class 'dict'>

TokenClassificationResult.to_dict[source]

TokenClassificationResult.to_dict(detail_level:DetailLevel='low')

Convert self to a dictionary

Parameters:

  • detail_level : <class 'fastcore.basics.DetailLevel'>, optional

    A level of detail to return

Returns:

  • <class 'dict'>

class TransformersTokenTagger[source]

TransformersTokenTagger(tokenizer:PreTrainedTokenizer, model:PreTrainedModel) :: AdaptiveModel

Adaptive model for Transformer's Token Tagger Model

Parameters:

  • tokenizer : <class 'transformers.tokenization_utils.PreTrainedTokenizer'>

    A tokenizer object from Huggingface's transformers (TODO) and tokenizers

  • model : <class 'transformers.modeling_utils.PreTrainedModel'>

    A transformers token tagger model

TransformersTokenTagger.load[source]

TransformersTokenTagger.load(model_name_or_path:str)

Class method for loading and constructing this tagger

Parameters:

  • model_name_or_path : <class 'str'>

    A key string of one of Transformer's pre-trained Token Tagger Model or a `HFModelResult`

Returns:

  • <class 'adaptnlp.model.AdaptiveModel'>

TransformersTokenTagger.predict[source]

TransformersTokenTagger.predict(text:Union[List[str], str], mini_batch_size:int=32, grouped_entities:bool=True, detail_level:DetailLevel='low', **kwargs)

Predict method for running inference using the pre-trained token tagger model

Parameters:

  • text : typing.Union[typing.List[str], str]

    Sentences to run inference on

  • mini_batch_size : <class 'int'>, optional

    Mini batch size

  • grouped_entities : <class 'bool'>, optional

    Return whole entity span strings

  • detail_level : <class 'fastcore.basics.DetailLevel'>, optional

    A level of detail to return

  • kwargs : <class 'inspect._empty'>

Returns:

  • typing.List[typing.List[typing.Dict]]

    Returns a list of lists of tagged entities

class FlairTokenTagger[source]

FlairTokenTagger(model_name_or_path:str) :: AdaptiveModel

Adaptive Model for Flair's Token Tagger

Parameters:

  • model_name_or_path : <class 'str'>

    A key string of one of Flair's pre-trained Token tagger Model, [link](https://huggingface.co/models?filter=flair)

preds[0]['labels']

[]

FlairTokenTagger.load[source]

FlairTokenTagger.load(model_name_or_path:str)

Class method for loading a constructing this tagger

Parameters:

  • model_name_or_path : <class 'str'>

    A key string of one of Flair's pre-trained Token tagger Model, [link](https://huggingface.co/models?filter=flair)

Returns:

  • <class 'adaptnlp.model.AdaptiveModel'>

FlairTokenTagger.predict[source]

FlairTokenTagger.predict(text:Union[List[Sentence], Sentence, List[str], str], mini_batch_size:int=32, raw:bool=False, **kwargs)

Predict method for running inference using the pre-trained token tagger model

Parameters:

  • text : typing.Union[typing.List[flair.data.Sentence], flair.data.Sentence, typing.List[str], str]

    Sentences to run inference on

  • mini_batch_size : <class 'int'>, optional

    Mini batch size

  • raw : <class 'bool'>, optional

    Whether to return a list of raw Sentences

  • kwargs : <class 'inspect._empty'>

Returns:

  • typing.List[flair.data.Sentence]

    A list of predicted sentences

class EasyTokenTagger[source]

EasyTokenTagger()

Token level classification models

tagger = EasyTokenTagger()
sentences = ["Jack walked through the park on a Sunday.", "Sunday was a nice and breezy afternoon.", "Jack was going to meet Jill for dinner."]
res = tagger.tag_text(text=sentences, model_name_or_path='flair/ner-english-ontonotes')
tagger.tag_text(text=sentences, model_name_or_path="flair/pos")
results = tagger.tag_all(sentences)

2021-10-05 20:24:53,169 loading file /root/.flair/models/ner-english-ontonotes-fast/0d55dd3b912da9cf26e003035a0c269a0e9ab222f0be1e48a3bbba3a58c0fed0.c9907cd5fde3ce84b71a4172e7ca03841cd81ab71d13eb68aa08b259f57c00b6
2021-10-05 20:24:57,389 loading file /root/.flair/models/pos-english-fast/36f7923039eed4c66e4275927daaff6cd275997d61d238355fb1fe0338fe10a1.ff87e5b4e47fdb42a0c00237d9506c671db773e0a7932179ace82e584383a1b8

EasyTokenTagger.tag_text[source]

EasyTokenTagger.tag_text(text:Union[List[Sentence], Sentence, List[str], str], model_name_or_path:Union[str, FlairModelResult, HFModelResult]='ner-ontonotes', mini_batch_size:int=32, detail_level:DetailLevel='low', **kwargs)

Tags tokens with labels the token classification models have been trained on

Parameters:

  • text : typing.Union[typing.List[flair.data.Sentence], flair.data.Sentence, typing.List[str], str]

    Text input, it can be a string or any of Flair's `Sentence` input formats

  • model_name_or_path : typing.Union[str, adaptnlp.model_hub.FlairModelResult, adaptnlp.model_hub.HFModelResult], optional

    The hosted model name key or model path

  • mini_batch_size : <class 'int'>, optional

    The mini batch size for running inference

  • detail_level : <class 'fastcore.basics.DetailLevel'>, optional

    The level fo detail to return on a TransformerTagger

  • kwargs : <class 'inspect._empty'>

Returns:

  • typing.List[flair.data.Sentence]

    A list of Flair's `Sentence`'s

EasyTokenTagger.tag_all[source]

EasyTokenTagger.tag_all(text:Union[List[Sentence], Sentence, List[str], str], mini_batch_size:int=32, detail_level:DetailLevel='low', **kwargs)

Tags tokens with all labels from all token classification models

Parameters:

  • text : typing.Union[typing.List[flair.data.Sentence], flair.data.Sentence, typing.List[str], str]

    Text input, it can be a string or any of Flair's `Sentence` input formats

  • mini_batch_size : <class 'int'>, optional

    The mini batch size for running inference

  • detail_level : <class 'fastcore.basics.DetailLevel'>, optional

    The level of detail for a TransformersTagger to return

  • kwargs : <class 'inspect._empty'>

Returns:

  • typing.List[dict]

    A dictionary of Token Tagging results