kiwi.data.fields package

Submodules

kiwi.data.fields.alignment_field module

class kiwi.data.fields.alignment_field.AlignmentField(sequential=True, use_vocab=True, init_token=None, eos_token=None, fix_length=None, dtype=torch.int64, preprocessing=None, postprocessing=None, lower=False, tokenize=<function Field.<lambda>>, include_lengths=False, batch_first=False, pad_token='<pad>', unk_token='<unk>', pad_first=False, truncate_first=False, stop_words=None, is_target=False)[source]

Bases: torchtext.data.field.Field

process(batch, *args, **kwargs)[source]

Process a list of examples to create a batch.

Postprocess the batch with user-provided Pipeline.

Parameters:batch (list(object)) – A list of object from a batch of examples.
Returns:
Processed object given the input and custom
postprocessing Pipeline.
Return type:object

kiwi.data.fields.qe_field module

class kiwi.data.fields.qe_field.QEField(unaligned_token='<unaligned>', unk_token='<unk>', pad_token='<pad>', init_token='<bos>', eos_token='<eos>', **kwargs)[source]

Bases: torchtext.data.field.Field

build_vocab(*args, **kwargs)[source]

Add unaligned_token to the list of special symbols.

kiwi.data.fields.sequence_labels_field module

class kiwi.data.fields.sequence_labels_field.SequenceLabelsField(classes, *args, **kwargs)[source]

Bases: torchtext.data.field.Field

Sequence of Labels.

build_vocab(*args, **kwargs)[source]

Construct the Vocab object for this field from one or more datasets.

Parameters:
  • arguments (Positional) – Dataset objects or other iterable data sources from which to construct the Vocab object that represents the set of possible values for this field. If a Dataset object is provided, all columns corresponding to this field are used; individual columns can also be provided directly.
  • keyword arguments (Remaining) – Passed to the constructor of Vocab.

Module contents