kiwi.modules.common.scorer

Module Contents

Classes

Scorer	Score function for attention module.
MLPScorer	MultiLayerPerceptron Scorer with variable nb of layers and neurons.

class kiwi.modules.common.scorer.Scorer(scaled: bool = True)

Bases: torch.nn.Module

Score function for attention module.

Parameters

scaled – whether to scale scores by sqrt(hidden_size) as proposed by the “Attention is All You Need” paper.

scale(self, hidden_size: int) → float

Denominator for scaling the scores.

Parameters

hidden_size – max hidden size between query and keys.

Returns

sqrt(hidden_size) if scaled is True, 1 otherwise.

abstract forward(self, query: torch.FloatTensor, keys: torch.FloatTensor) → torch.FloatTensor

Compute scores for each key of size n given the queries of size m.

The three dots (…) represent any other dimensions, such as the number of heads (useful if you use a multi head attention).

Parameters

• query – query matrix (bs, ..., target_len, m).

• keys – keys matrix (bs, ..., source_len, n).

Returns

matrix representing scores between source words and target words (bs, ..., target_len, source_len)

class kiwi.modules.common.scorer.MLPScorer(query_size, key_size, layer_sizes=None, activation=nn.Tanh, **kwargs)

Bases: kiwi.modules.common.scorer.Scorer

MultiLayerPerceptron Scorer with variable nb of layers and neurons.

forward(self, query, keys)

Compute scores for each key of size n given the queries of size m.

The three dots (…) represent any other dimensions, such as the number of heads (useful if you use a multi head attention).

Parameters

• query – query matrix (bs, ..., target_len, m).

• keys – keys matrix (bs, ..., source_len, n).

Returns

matrix representing scores between source words and target words (bs, ..., target_len, source_len)

kiwi.modules.common.scalar_mix kiwi.modules.sentence_level_output