Package opennlp.tools.util.featuregen
Class TokenPatternFeatureGenerator
java.lang.Object
opennlp.tools.util.featuregen.TokenPatternFeatureGenerator
- All Implemented Interfaces:
AdaptiveFeatureGenerator
Partitions tokens into sub-tokens based on character classes and generates
class features for each of the sub-tokens and combinations of those sub-tokens.
- See Also:
-
Constructor Summary
ConstructorsConstructorDescriptionInitializes aTokenPatternFeatureGenerator.TokenPatternFeatureGenerator(Tokenizer supportTokenizer) Initializes aTokenPatternFeatureGeneratorinstance. -
Method Summary
Modifier and TypeMethodDescriptionvoidcreateFeatures(List<String> feats, String[] toks, int index, String[] preds) Adds the appropriate features for the token at the specifiedindexwith the specified array ofpreviousOutcomesto the specified list of features.Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface opennlp.tools.util.featuregen.AdaptiveFeatureGenerator
clearAdaptiveData, updateAdaptiveData
-
Constructor Details
-
TokenPatternFeatureGenerator
public TokenPatternFeatureGenerator()Initializes aTokenPatternFeatureGenerator. For tokenization theSimpleTokenizeris used. -
TokenPatternFeatureGenerator
Initializes aTokenPatternFeatureGeneratorinstance.- Parameters:
supportTokenizer- TheTokenizerto be used.
-
-
Method Details
-
createFeatures
Description copied from interface:AdaptiveFeatureGeneratorAdds the appropriate features for the token at the specifiedindexwith the specified array ofpreviousOutcomesto the specified list of features.- Specified by:
createFeaturesin interfaceAdaptiveFeatureGenerator- Parameters:
feats- The list of features to be added to.toks- The tokens of the sentence or other text unit being processed.index- The index of the token which is currently being processed.preds- The outcomes for the tokens prior to the specified index.
-