kiwi.metrics package

Submodules

kiwi.metrics.functions module

kiwi.metrics.functions.confusion_matrix(hat_y, y, n_classes=None)

kiwi.metrics.functions.delta_average(y_true, y_rank)

Calculate the DeltaAvg score

This is a much faster version than the Perl one provided in the WMT QE task 1.

References: could not find any.

Author: Fabio Kepler (contributed to MARMOT)

Parameters:

• y_true – array of reference score (not rank) of each segment.
• y_rank – array of rank of each segment.

Returns: the absolute delta average score.

kiwi.metrics.functions.f1_product(hat_y, y)

kiwi.metrics.functions.f1_scores(hat_y, y)

Return f1_bad, f1_ok and f1_product

kiwi.metrics.functions.fscore(tp, fp, fn)

kiwi.metrics.functions.mean_absolute_error(y, y_hat)

kiwi.metrics.functions.mean_squared_error(y, y_hat)

kiwi.metrics.functions.precision(tp, fp, fn)

kiwi.metrics.functions.precision_recall_fscore_support(hat_y, y, labels=None)

kiwi.metrics.functions.recall(tp, fp, fn)

kiwi.metrics.functions.scores_for_class(class_index, cnfm)

kiwi.metrics.metrics module

class kiwi.metrics.metrics.CorrectMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(model_out, batch, **kwargs)

class kiwi.metrics.metrics.ExpectedErrorMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(model_out, batch, **kwargs)

class kiwi.metrics.metrics.F1Metric(labels, **kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(model_out, batch, **kwargs)

class kiwi.metrics.metrics.LogMetric(targets, metric_name=None, **kwargs)

Bases: kiwi.metrics.metrics.Metric

Logs averages of values in loss, model or batch.

reset()

summarize()

update(**kwargs)

class kiwi.metrics.metrics.Metric(target_name=None, metric_name=None, PAD=None, STOP=None, prefix=None)

Bases: object

get_name()

get_predictions(model_out)

get_predictions_flat(model_out, batch)

get_target(batch)

get_target_flat(batch)

get_token_indices(batch)

get_tokens(batch)

reset()

summarize(**kwargs)

token_mask(batch)

update(**kwargs)

class kiwi.metrics.metrics.MovingF1

Bases: kiwi.metrics.metrics.MovingMetric

choose(thresholds)

Choose the best (threshold, metric) tuple from an iterable.

compute()

Compute the current Value of the metric

init(scores, labels, class_idx=1)

Compute F1 Mult for all decision thresholds over (scores, labels) Initialize the threshold s.t. all examples are classified as class_idx. :param scores: Likelihood scores for class index :param Labels: Gold Truth classes in {0,1} :param class_index: ID of class

update(score, label)

Move the decision threshold.

class kiwi.metrics.metrics.MovingMetric

Bases: object

Class to compute the changes in one metric as a function of a second metric. Example: F1 score vs. Classification Threshold, Quality vs Skips

choose(thresholds)

Choose the best (threshold, metric) tuple from an iterable.

compute()

Compute the current Value of the metric

eval(scores, labels)

Compute the graph metric1 vs metric2 :param Scores: Model Outputs :param Labels: Corresponding Labels

init(scores, labels)

Initialize the Metric for threshold < min(scores)

sort(scores, labels)

Sort List of labels and scores.

update(score, label)

Move the threshold past score

class kiwi.metrics.metrics.MovingSkipsAtQuality(scores_higher_is_better=False, labels_higher_is_better=False)

Bases: kiwi.metrics.metrics.MovingMetric

Computes Quality of skipped examples vs fraction of skips.

choose(thresholds, target_qual)

Chooses the smallest threshold such that avg. quality is greater than or equal to target_qual

compute()

Compute the current Value of the metric

eval(scores, labels)

Parameters:

• scores – Model output quality or error scores. If quality scores are provided, pass scores_higher_is_better=True.
• labels – Ground truth quality or error scores. If quality scores are provided, pass labels_higher_is_better=True.

init(scores, labels)

Parameters:

• scores – Model output quality or error scores. If quality scores are provided, pass scores_higher_is_better=True.
• labels – Ground truth quality or error scores. If quality scores are provided, pass labels_higher_is_better=True.

sort(scores, labels)

Sort List of labels and scores.

update(score, label)

Move the threshold past score

class kiwi.metrics.metrics.NLLMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(loss, batch, **kwargs)

class kiwi.metrics.metrics.PearsonMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(model_out, batch, **kwargs)

class kiwi.metrics.metrics.PerplexityMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(loss, batch, **kwargs)

class kiwi.metrics.metrics.RMSEMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(batch, model_out, **kwargs)

class kiwi.metrics.metrics.SpearmanMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(model_out, batch, **kwargs)

class kiwi.metrics.metrics.ThresholdCalibrationMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(model_out, batch, **kwargs)

class kiwi.metrics.metrics.TokPerSecMetric(**kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(batch, **kwargs)

class kiwi.metrics.metrics.TokenMetric(target_token=0, token_name='UNK', **kwargs)

Bases: kiwi.metrics.metrics.Metric

reset()

summarize()

update(batch, **kwargs)

kiwi.metrics.stats module

class kiwi.metrics.stats.Stats(metrics, main_metric=None, main_metric_ordering=<built-in function max>, log_interval=0)

Bases: object

log(step=None)

reset()

summarize(prefix=None)

update(**kwargs)

wrap_up(prefix=None)

class kiwi.metrics.stats.StatsSummary(prefix=None, main_metric=None, ordering=<built-in function max>, **kwargs)

Bases: collections.OrderedDict

better_than(other)

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

log()

Log statistics to output and also to tracking logger.

Parameters:stats_summary – StatsSummary object

main_metric

main_metric_value()

Module contents