Package opennlp.tools.tokenize

Class WordpieceTokenizer

java.lang.Object

opennlp.tools.tokenize.WordpieceTokenizer

All Implemented Interfaces:

Tokenizer

public class WordpieceTokenizer
extends Object
implements Tokenizer

A Tokenizer implementation which performs tokenization using word pieces.

Adapted under MIT license from https://github.com/robrua/easy-bert.

Note that this tokenizer performs only the wordpiece (subword) stage of BERT tokenization. It does not normalize the input text: no lower casing, no accent stripping, no control character removal. Text that does not match the vocabulary's casing - for uncased models that includes every capitalized word - is mapped to the unknown token. Use BertTokenizer for the full BERT tokenization pipeline.

As of OpenNLP 3.0.0 the behavior matches the reference BERT wordpiece implementation in three respects that differ from earlier releases: runs of punctuation (and non-ASCII punctuation) are split into individual single-character tokens, words that cannot be fully represented by vocabulary pieces become a single unknown token instead of the matched prefix pieces followed by the unknown token, and tokenizePos(String) throws UnsupportedOperationException instead of returning null.

For reference see:

• https://www.tensorflow.org/text/guide/subwords_tokenizer#applying_wordpiece
• https://cran.r-project.org/web/packages/wordpiece/vignettes/basic_usage.html

See Also:

• BertTokenizer

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BERT_CLS_TOKEN	BERT classification token: [CLS].
static final String	BERT_SEP_TOKEN	BERT separator token: [SEP].
static final String	BERT_UNK_TOKEN	BERT unknown token: [UNK].
static final String	ROBERTA_CLS_TOKEN	RoBERTa classification token: <s>.
static final String	ROBERTA_SEP_TOKEN	RoBERTa separator token.
static final String	ROBERTA_UNK_TOKEN	RoBERTa unknown token.

Constructor Summary

Constructors

Constructor	Description
WordpieceTokenizer(Set<String> vocabulary)	Initializes a WordpieceTokenizer with a vocabulary and a default maxTokenLength of 50.
WordpieceTokenizer(Set<String> vocabulary, int maxTokenLength)	Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.
WordpieceTokenizer(Set<String> vocabulary, String classificationToken, String separatorToken, String unknownToken)	Initializes a WordpieceTokenizer with a vocabulary and custom special tokens.
WordpieceTokenizer(Set<String> vocabulary, String classificationToken, String separatorToken, String unknownToken, int maxTokenLength)	Initializes a WordpieceTokenizer with a vocabulary, custom special tokens and a custom maxTokenLength.

Method Summary

Modifier and Type	Method	Description
int	getMaxTokenLength()
String[]	tokenize(String text)	Splits a string into its atomic parts.
Span[]	tokenizePos(String text)	Not supported: wordpiece tokens (subwords, ## continuations and special tokens) have no faithful character spans in the original text.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

BERT_CLS_TOKEN

public static final String BERT_CLS_TOKEN

BERT classification token: [CLS].

See Also:

• Constant Field Values

BERT_SEP_TOKEN

public static final String BERT_SEP_TOKEN

BERT separator token: [SEP].

See Also:

• Constant Field Values

BERT_UNK_TOKEN

public static final String BERT_UNK_TOKEN

BERT unknown token: [UNK].

See Also:

• Constant Field Values

ROBERTA_CLS_TOKEN

public static final String ROBERTA_CLS_TOKEN

RoBERTa classification token: <s>.

See Also:

• Constant Field Values

ROBERTA_SEP_TOKEN

public static final String ROBERTA_SEP_TOKEN

RoBERTa separator token.

See Also:

• Constant Field Values

ROBERTA_UNK_TOKEN

public static final String ROBERTA_UNK_TOKEN

RoBERTa unknown token.

See Also:

• Constant Field Values

Constructor Details

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary)

Initializes a WordpieceTokenizer with a vocabulary and a default maxTokenLength of 50.

Parameters:

vocabulary - A set of tokens considered the vocabulary.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 int maxTokenLength)

Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.

Parameters:

vocabulary - A set of tokens considered the vocabulary.

maxTokenLength - A non-negative number that is used as maximum token length.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 String classificationToken,
 String separatorToken,
 String unknownToken)

Initializes a WordpieceTokenizer with a vocabulary and custom special tokens. This allows support for models like RoBERTa that use different special tokens instead of the BERT defaults.

Parameters:

vocabulary - The vocabulary.

classificationToken - The CLS token.

separatorToken - The SEP token.

unknownToken - The UNK token.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 String classificationToken,
 String separatorToken,
 String unknownToken,
 int maxTokenLength)

Initializes a WordpieceTokenizer with a vocabulary, custom special tokens and a custom maxTokenLength.

Parameters:

vocabulary - The vocabulary.

classificationToken - The CLS token.

separatorToken - The SEP token.

unknownToken - The UNK token.

maxTokenLength - A non-negative number that is used as maximum token length.

Method Details

tokenizePos

public Span[] tokenizePos(String text)

Not supported: wordpiece tokens (subwords, ## continuations and special tokens) have no faithful character spans in the original text.

Specified by:

tokenizePos in interface Tokenizer

Parameters:

text - The string to be tokenized.

Returns:

The spans (offsets into s) for each token as the individuals array elements.

Throws:

UnsupportedOperationException - Always.

tokenize

public String[] tokenize(String text)

Description copied from interface: Tokenizer

Splits a string into its atomic parts.

Specified by:

tokenize in interface Tokenizer

Parameters:

text - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.

getMaxTokenLength

public int getMaxTokenLength()

Returns:

The maximum token length.