Evaluate Interface

Note:Args that start with ‘--’ (eg. --save-config) can also be set in a config file (specified via --config). The config file uses YAML syntax and must represent a YAML ‘mapping’ (for details, see http://learn.getgrav.org/advanced/yaml). If an arg is specified in more than one place, then command line values override config file values which override defaults.

usage: kiwi evaluate [-h] [--type {probs,tags}] [--format {wmt17,wmt18}]
                     [--pred-format {wmt17,wmt18}] [--sents-avg {probs,tags}]
                     [--gold-sents GOLD_SENTS] [--gold-target GOLD_TARGET]
                     [--gold-source GOLD_SOURCE] [--gold-cal GOLD_CAL]
                     [--input-dir INPUT_DIR [INPUT_DIR ...]]
                     [--pred-sents PRED_SENTS [PRED_SENTS ...]]
                     [--pred-target PRED_TARGET [PRED_TARGET ...]]
                     [--pred-gaps PRED_GAPS [PRED_GAPS ...]]
                     [--pred-source PRED_SOURCE [PRED_SOURCE ...]]
                     [--pred-cal PRED_CAL]

Evaluation of WMT Quality Estimation

--type	Possible choices: probs, tags Input type for prediction file Default: “probs”
--format	Possible choices: wmt17, wmt18 Input format for gold files Default: “wmt17”
--pred-format	Possible choices: wmt17, wmt18 Input format for predicted files. Defaults to the same as –format. Default: “wmt18”
--sents-avg	Possible choices: probs, tags Obtain scores for sentences by averaging over tags or probabilities.
--gold-sents	Sentences gold standard.
--gold-target	Target tags gold standard, or target and gaps if format == “wmt18”.
--gold-source	Source tags gold standard.
--gold-cal	Target Tags to calibrate.
--input-dir	Directory with prediction files generated by predict pipeline. Setting this argument will evaluate all predictions for which a gold file is set.
--pred-sents	Sentences HTER predictions.
--pred-target	Target predictions; can be tags or probabilities (of BAD). See –type.
--pred-gaps	Gap predictions; can be tags or probabilities (of BAD). (see –type). Use this option for files that only contain gap tags.
--pred-source	Source predictions. can be tags or probabilities (of BAD). See –type.
--pred-cal	Target Predictions to calibrate.