cc.twittertools.corpus.demo
Class LowerCaseHashtagMentionPreservingTokenizer

java.lang.Object
  org.apache.lucene.util.AttributeSource
      org.apache.lucene.analysis.TokenStream
          org.apache.lucene.analysis.Tokenizer
              org.apache.lucene.analysis.CharTokenizer
                  org.apache.lucene.analysis.LetterTokenizer
                      cc.twittertools.corpus.demo.LowerCaseHashtagMentionPreservingTokenizer

All Implemented Interfaces:

Closeable

public class LowerCaseHashtagMentionPreservingTokenizer
extends org.apache.lucene.analysis.LetterTokenizer

A variant of Lucene's LowerCaseTokenizer, but preserves hashtags and mentions.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Constructor Summary

LowerCaseHashtagMentionPreservingTokenizer(org.apache.lucene.util.Version matchVersion, Reader in)

Method Summary

Methods inherited from class org.apache.lucene.analysis.CharTokenizer

end, incrementToken, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close

Methods inherited from class org.apache.lucene.analysis.TokenStream

reset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

LowerCaseHashtagMentionPreservingTokenizer

public LowerCaseHashtagMentionPreservingTokenizer(org.apache.lucene.util.Version matchVersion,
                                                  Reader in)