Uses of Class
net.sf.okapi.steps.tokenization.tokens.Tokens

Packages that use Tokens

Package	Description
net.sf.okapi.steps.tokenization
net.sf.okapi.steps.tokenization.common
net.sf.okapi.steps.tokenization.engine
net.sf.okapi.steps.tokenization.tokens
net.sf.okapi.steps.wordcount.categorized.gmx
net.sf.okapi.steps.wordcount.common

Uses of Tokens in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Tokens

Modifier and Type	Method and Description
static Tokens	Tokenizer.tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	Tokenizer.tokenizeString(String text, LocaleId language, String... tokenNames) Extracts tokens from the given text.

Uses of Tokens in net.sf.okapi.steps.tokenization.common

Methods in net.sf.okapi.steps.tokenization.common that return Tokens

Modifier and Type	Method and Description
Tokens	TokensAnnotation.getFilteredList(String... tokenTypes)
Tokens	TokensAnnotation.getTokens()

Methods in net.sf.okapi.steps.tokenization.common with parameters of type Tokens

Modifier and Type	Method and Description
void	TokensAnnotation.addTokens(Tokens tokens)
protected abstract void	AbstractLexer.lexer_open(String text, LocaleId language, Tokens tokens)
void	ILexer.open(String text, LocaleId language, Tokens tokens) Starts processing a string or a list of tokens, extracting lexems from them.
void	AbstractLexer.open(String text, LocaleId language, Tokens tokens)
Lexems	ILexer.process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.
void	TokensAnnotation.setTokens(Tokens tokens)

Constructors in net.sf.okapi.steps.tokenization.common with parameters of type Tokens

Constructor and Description
TokensAnnotation(Tokens tokens)

Uses of Tokens in net.sf.okapi.steps.tokenization.engine

Methods in net.sf.okapi.steps.tokenization.engine with parameters of type Tokens

Modifier and Type	Method and Description
protected void	TokenScanner.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	TokenMatcher.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	TextScanner.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	Sorter.lexer_open(String text, LocaleId language, Tokens tokens)
void	Retainer.lexer_open(String text, LocaleId language, Tokens tokens)
void	Remover.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	Reconciler.lexer_open(String text, LocaleId language, Tokens tokens)
void	RbbiLexer.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	LookUpLexer.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	JavaCcLexer.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	GarbageCollector.lexer_open(String text, LocaleId language, Tokens tokens)
Lexems	TokenScanner.process(String text, LocaleId language, Tokens tokens)
Lexems	TokenMatcher.process(String text, LocaleId language, Tokens tokens)
Lexems	TextScanner.process(String text, LocaleId language, Tokens tokens)
Lexems	Sorter.process(String text, LocaleId language, Tokens tokens)
Lexems	Retainer.process(String text, LocaleId language, Tokens tokens)
Lexems	Remover.process(String text, LocaleId language, Tokens tokens)
Lexems	Reconciler.process(String text, LocaleId language, Tokens tokens)
Lexems	RbbiLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	LookUpLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	JavaCcLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	GarbageCollector.process(String text, LocaleId language, Tokens tokens)

Uses of Tokens in net.sf.okapi.steps.tokenization.tokens

Methods in net.sf.okapi.steps.tokenization.tokens that return Tokens

Modifier and Type	Method and Description
Tokens	Tokens.getFilteredList(String... tokenNames) Return a list of Token objects.

Uses of Tokens in net.sf.okapi.steps.wordcount.categorized.gmx

Methods in net.sf.okapi.steps.wordcount.categorized.gmx that return Tokens

Modifier and Type	Method and Description
protected Tokens	GMXNumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXNumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)

Methods in net.sf.okapi.steps.wordcount.categorized.gmx with parameters of type Tokens

Modifier and Type	Method and Description
protected Tokens	GMXNumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXNumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXMeasurementOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitWordCountStep.filterTokens(Tokens allTokens)
protected Tokens	GMXAlphanumericOnlyTextUnitCharacterCountStep.filterTokens(Tokens allTokens)

Uses of Tokens in net.sf.okapi.steps.wordcount.common

Methods in net.sf.okapi.steps.wordcount.common that return Tokens

Modifier and Type	Method and Description
protected Tokens	TokenCountStep.filterTokens(Tokens allTokens)
protected Tokens	TokenCountStep.getTokens(Segment segment, LocaleId locale)
protected Tokens	TokenCountStep.getTokens(TextContainer textContainer, LocaleId locale)

Methods in net.sf.okapi.steps.wordcount.common with parameters of type Tokens

Modifier and Type	Method and Description
static long	TokenCharacterCountStep.countTokenChars(Tokens tokens, LocaleId locale) Return the total character count (calculated per GMX.TotalCharacterCount) of all supplied tokens.
protected Tokens	TokenCountStep.filterTokens(Tokens allTokens)