org.languagetool.tokenizers

Interface Tokenizer

All Known Subinterfaces:

CompoundWordTokenizer, SentenceTokenizer

All Known Implementing Classes:

SimpleSentenceTokenizer, SRXSentenceTokenizer, WordTokenizer

public interface Tokenizer

Interface for classes that tokenize text into smaller units.

Author:

Daniel Naber

Method Summary

Modifier and Type	Method and Description
List<String>	tokenize(String text)

Method Detail

tokenize

List<String> tokenize(String text)