Squad Metrics from the Transformers library

normalize_answer[source]

normalize_answer(s)

Lower text and remove punctuation, articles and extra whitespace.

get_tokens[source]

get_tokens(s)

compute_exact[source]

compute_exact(a_gold, a_pred)

compute_f1[source]

compute_f1(a_gold, a_pred)

get_raw_scores[source]

get_raw_scores(examples, preds)

Computes the exact and f1 scores from the examples and the model predictions

apply_no_ans_threshold[source]

apply_no_ans_threshold(scores, na_probs, qid_to_has_ans, na_prob_thresh)

make_eval_dict[source]

make_eval_dict(exact_scores, f1_scores, qid_list=None)

merge_eval[source]

merge_eval(main_eval, new_eval, prefix)

find_best_thresh_v2[source]

find_best_thresh_v2(preds, scores, na_probs, qid_to_has_ans)

find_all_best_thresh_v2[source]

find_all_best_thresh_v2(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans)

find_best_thresh[source]

find_best_thresh(preds, scores, na_probs, qid_to_has_ans)

find_all_best_thresh[source]

find_all_best_thresh(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans)

squad_evaluate[source]

squad_evaluate(examples, preds, no_answer_probs=None, no_answer_probability_threshold=1.0)

Usage Examples:

Sample code from: 13c_transformers.utils_squad_evaluate.ipynb (View Notebook for more context)


get_final_text[source]

get_final_text(pred_text, orig_text, do_lower_case, verbose_logging=False)

Project the tokenized prediction back to the original text.

compute_predictions_logits[source]

compute_predictions_logits(all_examples, all_features, all_results, n_best_size, max_answer_length, do_lower_case, verbose_logging, version_2_with_negative, null_score_diff_threshold, tokenizer, output_prediction_file=None, output_nbest_file=None, output_null_log_odds_file=None)

compute_predictions_log_probs[source]

compute_predictions_log_probs(all_examples, all_features, all_results, n_best_size, max_answer_length, start_n_top, end_n_top, version_2_with_negative, tokenizer, verbose_logging, output_prediction_file=None, output_nbest_file=None, output_null_log_odds_file=None)

XLNet write prediction logic (more complex than Bert's). Write final predictions to the json file and log-odds of null if needed. Requires utils_squad_evaluate.py