Uses of Interface
opennlp.tools.tokenize.Tokenizer

Packages that use Tokenizer

Package	Description
opennlp.tools.tokenize

Uses of Tokenizer in opennlp.tools.tokenize

Classes in opennlp.tools.tokenize that implement Tokenizer

Modifier and Type	Class	Description
class	BertTokenizer	A Tokenizer implementation of the full BERT tokenization pipeline: basic tokenization (text normalization) followed by wordpiece tokenization.
class	WhitespaceTokenizer	A basic Tokenizer implementation which performs tokenization using white spaces.
class	WordpieceTokenizer	A Tokenizer implementation which performs tokenization using word pieces.