Quickstart

This document gives a quick overview of how to use OpenKiwi with the provided example configuration file.

New: We have released a tutorial that presents basic usage of OpenKiwi. It is available here.

To be able to run the examples, download the WMT17 En-DE Word Level Quality Estimation data (you will need to provide your email address to receive a download link) and extract its contents into data/WMT17. To be able to pretrain the predictor model, additionally download the English-German in-domain Corpus provided by WMT and extract its contents into data/WMT18.

OpenKiwi can be used as a package from within Python or from the command line.

Its functionality is split in pipelines. Currently supported are:

train, predict, jackknife, evaluate

The models supported are:

The Estimator model relies on pre-training of its component model predictor.

Options are handled via yaml configuration files. On the command line, you can alternatively pass parameters as arguments, or mix the two modes to override parameters in a config file.

Train Pipeline

Before we continue into specific instructions for training models please take into account that if you’re working with data from WMT18 or later, you must make sure to pass the --wmt18-format flag. Either through yaml, or the CLI. This happens because WMT18 introduced GAP tags that classify the gaps between words. Note that the example files provided use WMT17 data for the predictor-estimator and WMT18 data for NuQE. You can check all the configuration options in here

Estimator Model

Example Configuration files with detailed hyperparameter documentation:

The Estimator Model performs best if initialized with a predictor model that is pretrained on parallel data. The example configuration file requires ca. 24hrs to train 3 epochs on a GPU with 12GB RAM. To train with less memory, change the train-batch-size and valid-batch-size parameters. To train on CPU, comment out the line

gpu-id: 0

Train the predictor like so:

import kiwi

predictor_config = experiments/train_predictor.yaml'
kiwi.train(predictor_config)

Or via CLI:

kiwi train --config experiments/train_predictor.yaml

After the run has finished, the best model is saved at (--output-dir option) runs/predictor/best_model.torch. Now we can train the estimator model using the configuration experiments/train_estimator.yaml:

import kiwi

estimator_config = 'experiments/train_estimator.yaml'
kiwi.train(estimator_config)

Or:

kiwi train --config experiments/train_estimator.yaml

When a pretrained predictor is provided in this way, the hyperparameters relating to the predictor architecture as well as the vocabulary mapping are retrieved from the model. Setting them to different values in the estimator config has no effect.

NuQE Model

Example Configuration file with detailed hyperparameter documentation:

Run via API:

import kiwi

nuqe_config = 'experiments/train_nuqe.yaml'
kiwi.train(nuqe_config)

Or via CLI:

kiwi train --config experiments/train_nuqe.yaml

Predict

The predict pipeline takes a trained QE model as input and uses it to evaluate the quality of machine translations. The API provides the function load_model which returns a Predictor object that can be used to generate predictions.

To predict the quality of a set of machine translation dataset via the CLI, use a yaml file specifying model path, output directory, source and target data, and run:

kiwi predict --config experiments/predict_predest.yaml

kiwi predict --config experiments/predict_nuqe.yaml

which will generate predictions for the word and sentence level quality of the WMT18 en-de dev set in the folder predictions/predest.

Evaluate

The evaluate pipeline takes predictions of a trained model and a reference (gold) file and evaluates the performance of the model based on the comparison between the predictions and the reference.

To evaluate one of your models via the CLI, create a yaml file specifying the format of predictions, format of reference and the location of these files, and run:

kiwi evaluate --config experiments/estimator_evaluate.yaml

kiwi evaluate --config experiments/nuqe_evaluate.yaml

You can check all the configuration options in here