kiwi.data.tokenizers

Module Contents

Functions

tokenize(sentence)	Implement your own tokenize procedure.
detokenize(tokens)
align_tokenize(s)	Return a list of pair of integers for each sentence.
bert_tokenizer(sentence)

kiwi.data.tokenizers.tokenize(sentence)

Implement your own tokenize procedure.

kiwi.data.tokenizers.detokenize(tokens)

kiwi.data.tokenizers.align_tokenize(s)

Return a list of pair of integers for each sentence.

kiwi.data.tokenizers.bert_tokenizer(sentence)

kiwi.data.batch kiwi.data.vocabulary