Package opennlp.tools.tokenize
Class DictionaryDetokenizer
java.lang.Object
opennlp.tools.tokenize.DictionaryDetokenizer
- All Implemented Interfaces:
Detokenizer
A rule based detokenizer. Simple rules which indicate in which direction a token should be
moved are looked up in a
dictionary.- See Also:
-
Nested Class Summary
Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer
Detokenizer.DetokenizationOperation -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptiondetokenize(String[] tokens) Detokenizes the collection of tokens.detokenize(String[] tokens, String splitMarker) Detokenizes the inputtokensinto a String.
-
Constructor Details
-
DictionaryDetokenizer
Initializes aDictionaryDetokenizerinstance.- Parameters:
dict- TheDetokenizationDictionaryto be used.
-
-
Method Details
-
detokenize
Description copied from interface:DetokenizerDetokenizes the collection of tokens.- Specified by:
detokenizein interfaceDetokenizer- Parameters:
tokens- The elements which should be detokenized.- Returns:
- The
merge operationsto handle giventokens.
-
detokenize
Description copied from interface:DetokenizerDetokenizes the inputtokensinto a String. Tokens which are connected without awhitespacecharacter in between can be separated by a givensplitMarker.- Specified by:
detokenizein interfaceDetokenizer- Parameters:
tokens- The elements which should be concatenated.splitMarker- The split marker ornull.- Returns:
- The concatenated tokens as a single string.
-