Uses of Class
net.sf.okapi.steps.tokenization.Tokens

Packages that use Tokens

Package	Description
net.sf.okapi.steps.tokenization
net.sf.okapi.steps.wordcount.categorized.gmx
net.sf.okapi.steps.wordcount.common

Uses of Tokens in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Tokens

Modifier and Type	Method and Description
Tokens	Tokens.getExcludedFilteredList(String... tokenNames) Return a list of TokenType objects.
Tokens	TokensAnnotation.getFilteredList(String... tokenTypes)
Tokens	Tokens.getFilteredList(String... tokenNames) Return a list of TokenType objects.
Tokens	TokensAnnotation.getTokens()
static Tokens	Tokenizer.tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	Tokenizer.tokenizeString(String text, LocaleId language, String... tokenNames) Extracts tokens from the given text.

Methods in net.sf.okapi.steps.tokenization with parameters of type Tokens

Modifier and Type	Method and Description
void	TokensAnnotation.addTokens(Tokens tokens)
void	TokensAnnotation.setTokens(Tokens tokens)

Constructors in net.sf.okapi.steps.tokenization with parameters of type Tokens

Constructor and Description
TokensAnnotation(Tokens tokens)

Uses of Tokens in net.sf.okapi.steps.wordcount.categorized.gmx

Methods in net.sf.okapi.steps.wordcount.categorized.gmx that return Tokens

Modifier and Type	Method and Description
protected Tokens	GMXNumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXNumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)

Methods in net.sf.okapi.steps.wordcount.categorized.gmx with parameters of type Tokens

Modifier and Type	Method and Description
protected Tokens	GMXNumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXNumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)

Uses of Tokens in net.sf.okapi.steps.wordcount.common

Methods in net.sf.okapi.steps.wordcount.common that return Tokens

Modifier and Type	Method and Description
protected Tokens	TokenCountStep.filterTokens(Tokens allTokens)
protected Tokens	TokenCountStep.getTokens(Segment segment, LocaleId locale)
protected Tokens	TokenCountStep.getTokens(TextContainer textContainer, LocaleId locale)

Methods in net.sf.okapi.steps.wordcount.common with parameters of type Tokens

Modifier and Type	Method and Description
static long	TokenCharacterCountStep.countTokenChars(Tokens tokens, LocaleId locale) Return the total character count (calculated per GMX.TotalCharacterCount) of all supplied tokens.
protected Tokens	TokenCountStep.filterTokens(Tokens allTokens)