kiwi.models.linear package

Submodules

kiwi.models.linear.label_dictionary module

This implements a dictionary of labels.

class kiwi.models.linear.label_dictionary.LabelDictionary(label_names=None)

Bases: dict

This class implements a dictionary of labels. Labels as mapped to integers, and it is efficient to retrieve the label name from its integer representation, and vice-versa.

add(name)

Add new label.

get_label_id(name)

Get label id from name.

get_label_name(label_id)

Get label name from id.

load(label_file)

Load labels from a file.

save(label_file)

Save labels to a file.

kiwi.models.linear.linear_model module

This implements a linear model.

class kiwi.models.linear.linear_model.LinearModel

Bases: object

An abstract linear model.

clear()

Clear all weights.

compute_score(features)

Compute a score by taking the inner product with a feature vector.

compute_score_binary_features(binary_features)

Compute a score by taking the inner product with a binary feature vector.

finalize(t)

Finalize by setting the weights as the running average. This is a no-op if use_average=False.

load(model_file, average=False, feature_indices=None)

Load the model from a file.

make_gradient_step(features, eta, t, gradient)

Make a gradient step with stepsize eta.

read_fnames(fnames_file)

Read file mapping from integers to feature descriptions.

save(model_file, average=False, feature_indices=None)

Save the model to a file.

write_fnames(fnames_file, fnames)

Write file mapping from integers to feature descriptions.

kiwi.models.linear.linear_trainer module

A generic implementation of a basic trainer.

class kiwi.models.linear.linear_trainer.LinearTrainer(classifier, checkpointer, algorithm='svm_mira', regularization_constant=1000000000000.0)

Bases: object

run(train_iterator, valid_iterator, epochs=50)

Train with a general online algorithm.

save(output_directory)

kiwi.models.linear.linear_word_qe_decoder module

Decoder for word-level quality estimation.

class kiwi.models.linear.linear_word_qe_decoder.LinearWordQEDecoder(estimator, cost_false_positives=0.5, cost_false_negatives=0.5)

Bases: kiwi.models.linear.structured_decoder.StructuredDecoder

A decoder for word-level quality estimation.

decode(instance, parts, scores)

Decoder. Return the most likely sequence of OK/BAD labels.

decode_mira(instance, parts, scores, gold_outputs, old_mira=False)

Cost-augmented decoder. Allows a compromise between precision and recall. In general: p = a - (a+b)*z0 q = b*sum(z0) p’*z + q = a*sum(z) - (a+b)*z0’*z + b*sum(z0)

= a*(1-z0)’*z + b*(1-z)’*z0

a => penalty for predicting 1 when it is 0 (FP) b => penalty for predicting 0 when it is 1 (FN)

F1: a = 0.5, b = 0.5 recall: a = 0, b = 1

decode_with_bigrams(instance, parts, scores)

Decoder for a sequential model (with bigrams).

decode_with_unigrams(instance, parts, scores)

Decoder for a non-sequential model (unigrams only).

run_viterbi(initial_scores, transition_scores, final_scores, emission_scores)

Computes the viterbi trellis for a given sequence. Receives: - Initial scores: (num_states) array - Transition scores: (length-1, num_states, num_states) array - Final scores: (num_states) array - Emission scores: (length, num_states) array.

kiwi.models.linear.linear_word_qe_decoder.logzero()

Return log of zero.

kiwi.models.linear.linear_word_qe_features module

A class for handling features for word-level quality estimation.

class kiwi.models.linear.linear_word_qe_features.LinearWordQEFeatures(use_basic_features_only=True, use_simple_bigram_features=True, use_parse_features=False, use_stacked_features=False, save_to_cache=False, load_from_cache=False, cached_features_file=None)

Bases: kiwi.models.linear.sparse_feature_vector.SparseFeatureVector

This class implements a feature vector for word-level quality estimation.

compute_bigram_features(sentence_word_features, part)

Compute bigram features (that depend on consecutive labels).

compute_unigram_features(sentence_word_features, part)

Compute unigram features (depending only on a single label).

get_head(sentence_word_features, index)

get_siblings(sentence_word_features, index)

kiwi.models.linear.linear_word_qe_features.quantize(value, bins_down)

Quantize a numeric feature into bins. Example: bins = [50, 40, 30, 25, 20, 18, 16, 14, 12, 10].

kiwi.models.linear.linear_word_qe_sentence module

class kiwi.models.linear.linear_word_qe_sentence.LinearWordQESentence

Bases: object

Represents a sentence (word features and their labels).

create_from_sentence_pair(source_words, target_words, alignments, source_pos_tags=None, target_pos_tags=None, target_parse_heads=None, target_parse_relations=None, target_ngram_left=None, target_ngram_right=None, target_stacked_features=None, labels=None)

Creates an instance from source/target token and alignment information.

static create_stop_symbol()

Generates dummy features for a stop symbol.

num_words()

Returns the number of words of the sentence.

class kiwi.models.linear.linear_word_qe_sentence.LinearWordQETokenFeatures(stacked_features=None, source_token_count=-1, target_token_count=-1, source_target_token_count_ratio=0.0, token='', left_context='', right_context='', first_aligned_token='', left_alignment='', right_alignment='', is_stopword=False, is_punctuation=False, is_proper_noun=False, is_digit=False, highest_order_ngram_left=-1, highest_order_ngram_right=-1, backoff_behavior_left=0.0, backoff_behavior_middle=0.0, backoff_behavior_right=0.0, source_highest_order_ngram_left=-1, source_highest_order_ngram_right=-1, pseudo_reference=False, target_pos='', target_morph='', target_head=-1, target_deprel='', aligned_source_pos_list='', polysemy_count_source=0, polysemy_count_target=0)

Bases: object

kiwi.models.linear.sequence_parts module

class kiwi.models.linear.sequence_parts.SequenceBigramPart(index, label, previous_label)

Bases: object

A part for bigrams (two labels at consecutive words position). Necessary for the model to be sequential.

class kiwi.models.linear.sequence_parts.SequenceUnigramPart(index, label)

Bases: object

A part for unigrams (a single label at a word position).

kiwi.models.linear.sparse_feature_vector module

This defines the class for defining sparse features in linear models.

class kiwi.models.linear.sparse_feature_vector.SparseBinaryFeatureVector(feature_indices=None, save_to_cache=False, load_from_cache=False, cached_features_file=None)

Bases: list

A generic class for a sparse binary feature vector.

add_binary_feature(name)

Add a binary feature.

add_categorical_feature(name, value)

Add a categorical feature, represented internally as a binary feature.

load_cached_features()

Load features from file.

save_cached_features()

Save features to file.

to_sparse_vector()

Convert to a SparseVector.

class kiwi.models.linear.sparse_feature_vector.SparseFeatureVector(save_to_cache=False, load_from_cache=False, cached_features_file=None)

Bases: kiwi.models.linear.sparse_vector.SparseVector

A generic class for a sparse feature vector.

add_binary_feature(name)

Add a binary feature.

add_categorical_feature(name, value, allow_duplicates=False)

Add a categorical feature, represented internally as a binary feature.

add_numeric_feature(name, value)

Add a numeric feature.

load_cached_features()

Load features from file.

save_cached_features()

Save features to file.

kiwi.models.linear.sparse_vector module

This defines a generic class for sparse vectors.

class kiwi.models.linear.sparse_vector.SparseVector

Bases: dict

Implementation of a sparse vector using a dictionary.

add(vector, scalar=1.0)

Adds this vector and a given vector.

add_constant(scalar)

Adds a constant to each element of the vector.

as_string()

Returns a string representation.

copy()

Returns a copy of the current vector.

dot_product(vector)

Computes the dot product with a given vector. Note: this iterates through the self vector, so it may be inefficient if the number of nonzeros in self is much larger than the number of nonzeros in vector. Hence the function reverts to vector.dot_product(self) if that is beneficial.

load(f, dtype=<class 'str'>)

Load vector from file.

normalize()

Normalize the vector. Note: if the norm is zero, do nothing.

save(f)

Save vector to file.

scale(scalar)

Scales this vector by a scale factor.

squared_norm()

Computes the squared norm of the vector.

kiwi.models.linear.structured_classifier module

A generic implementation of an abstract structured linear classifier.

class kiwi.models.linear.structured_classifier.StructuredClassifier

Bases: object

An abstract structured classifier.

compute_scores(instance, parts, features)

Compute a score for every part in the instance using the current model and the part-specific features.

create_instances(dataset)

Preprocess the dataset if needed to create instances. Default is returning the dataset itself. Override if needed.

create_prediction(instance, parts, predicted_output)

Create a prediction for an instance.

evaluate(instances, predictions, print_scores=True)

Evaluate the structure classifier, computing a task-dependent evaluation metric.

label_instance(instance, parts, predicted_output)

Return a labeled instance by adding the predicted output information.

load(model_path)

Load the full configuration and model.

make_features(instance, parts)

Create a feature vector for each part.

make_parts(instance)

Compute the task-specific parts for this instance.

run(instance)

Run the structured classifier on a single instance.

save(model_path)

Save the full configuration and model.

test(instances)

Run the structured classifier on dev/test data.

kiwi.models.linear.structured_decoder module

class kiwi.models.linear.structured_decoder.StructuredDecoder

Bases: object

An abstract decoder for structured prediction.

decode(instance, parts, scores)

Decode, computing the highest-scores output. Must return a vector of 0/1 predicted_outputs of the same size as parts.

decode_cost_augmented(instance, parts, scores, gold_outputs)

Perform cost-augmented decoding.

decode_mira(instance, parts, scores, gold_outputs, old_mira=False)

Perform cost-augmented decoding or classical MIRA.

kiwi.models.linear.utils module

Several utility functions.

kiwi.models.linear.utils.nearly_binary_tol(a, tol)

Checks if a number is binary up to a tolerance.

kiwi.models.linear.utils.nearly_eq_tol(a, b, tol)

Checks if two numbers are equal up to a tolerance.

kiwi.models.linear.utils.nearly_zero_tol(a, tol)

Checks if a number is zero up to a tolerance.

Module contents