Package opennlp.spellcheck.normalizer

Class SpellCheckingCharSequenceNormalizer

java.lang.Object

opennlp.spellcheck.normalizer.SpellCheckingCharSequenceNormalizer

All Implemented Interfaces:

Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer

public class SpellCheckingCharSequenceNormalizer
extends Object
implements opennlp.tools.util.normalizer.CharSequenceNormalizer

A CharSequenceNormalizer that corrects spelling in text using a SpellChecker (typically a SymSpell engine).

The normalizer works in one of two modes:

• PER_TOKEN (default) – the input is split into whitespace-delimited tokens and each token is corrected independently with SpellChecker.lookup(java.lang.String, opennlp.spellcheck.Verbosity, int). The original whitespace runs between tokens are preserved verbatim, so the shape of the line is kept. Tokens the dictionary already contains (best suggestion at edit distance 0) are left untouched.
• COMPOUND – the whole input is passed to SpellChecker.lookupCompound(java.lang.String, int), which additionally repairs wrongly inserted or omitted spaces (word splits and merges). This collapses runs of whitespace to single spaces, as the compound corrector re-tokenizes the input.

Several guards keep the corrector from "fixing" tokens that should be left as they are (configurable through the SpellCheckingCharSequenceNormalizer.Builder):

• tokens shorter than minTokenLength are skipped;
• numeric tokens are skipped (skipNumbers, on by default);
• URL- and email-like tokens are skipped (skipUrls, on by default);
• a token whose lower-cased form is already in the dictionary is never changed (the engine returns it at edit distance 0).

Casing. Dictionaries are normally lower-cased, so lookups are performed on the lower-cased token, and the original casing pattern is re-applied to the correction: an all-upper token yields an all-upper correction, a leading-capital token yields a leading-capital correction, otherwise the suggestion's own casing is used. When no correction applies, the original token (including its casing and any surrounding punctuation) is emitted unchanged.

This normalizer composes cleanly inside an AggregateCharSequenceNormalizer; place it after noise-removing normalizers (URL, emoji, shrink) so it sees clean tokens.

Serialization. CharSequenceNormalizer is Serializable, but the backing SpellChecker usually is not; it is therefore held in a transient field and is null after Java deserialization. A deserialized instance is inert until a checker is re-attached: obtain a working copy with the same settings via withSpellChecker(SpellChecker) (this matches how the engine is rebuilt from a model rather than Java-serialized). Calling normalize(java.lang.CharSequence) on an instance with no checker throws IllegalStateException.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	SpellCheckingCharSequenceNormalizer.Builder	A mutable builder for SpellCheckingCharSequenceNormalizer.
static enum	SpellCheckingCharSequenceNormalizer.Mode	The correction mode.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MIN_TOKEN_LENGTH	The default minimum token length below which tokens are left untouched.

Constructor Summary

Constructors

Constructor	Description
SpellCheckingCharSequenceNormalizer(SymSpellModel model)	Creates a normalizer in SpellCheckingCharSequenceNormalizer.Mode.PER_TOKEN mode with default guards from a loaded SymSpellModel (uses the model's engine).
SpellCheckingCharSequenceNormalizer(SpellChecker spellChecker)	Creates a normalizer in SpellCheckingCharSequenceNormalizer.Mode.PER_TOKEN mode with default guards from a SpellChecker.

Method Summary

Modifier and Type	Method	Description
static SpellCheckingCharSequenceNormalizer.Builder	builder(SymSpellModel model)
static SpellCheckingCharSequenceNormalizer.Builder	builder(SpellChecker spellChecker)
CharSequence	normalize(CharSequence text)
SpellCheckingCharSequenceNormalizer	withSpellChecker(SpellChecker checker)	Returns a copy of this normalizer carrying the same settings but backed by the given checker.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MIN_TOKEN_LENGTH

public static final int DEFAULT_MIN_TOKEN_LENGTH

The default minimum token length below which tokens are left untouched.

See Also:

• Constant Field Values

Constructor Details

SpellCheckingCharSequenceNormalizer

public SpellCheckingCharSequenceNormalizer(SpellChecker spellChecker)

Creates a normalizer in SpellCheckingCharSequenceNormalizer.Mode.PER_TOKEN mode with default guards from a SpellChecker.

Parameters:

spellChecker - the engine used to correct tokens; must not be null

SpellCheckingCharSequenceNormalizer

public SpellCheckingCharSequenceNormalizer(SymSpellModel model)

Creates a normalizer in SpellCheckingCharSequenceNormalizer.Mode.PER_TOKEN mode with default guards from a loaded SymSpellModel (uses the model's engine).

Parameters:

model - the loaded model whose engine is used; must not be null

Method Details

builder

public static SpellCheckingCharSequenceNormalizer.Builder builder(SpellChecker spellChecker)

Parameters:

spellChecker - the engine to wrap; must not be null

Returns:

a new SpellCheckingCharSequenceNormalizer.Builder seeded with sensible defaults

builder

public static SpellCheckingCharSequenceNormalizer.Builder builder(SymSpellModel model)

Parameters:

model - the loaded model whose engine to wrap; must not be null

Returns:

a new SpellCheckingCharSequenceNormalizer.Builder seeded with sensible defaults

withSpellChecker

public SpellCheckingCharSequenceNormalizer withSpellChecker(SpellChecker checker)

Returns a copy of this normalizer carrying the same settings but backed by the given checker. This is the supported way to re-attach an engine to an instance restored by Java deserialization (whose transient checker is null).

Parameters:

checker - the engine to attach; must not be null

Returns:

a new, ready-to-use normalizer with this instance's settings

normalize

public CharSequence normalize(CharSequence text)

Specified by:

normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer