org.apache.solr.analysis
Class RemoveDuplicatesTokenFilter

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.TokenFilter
              extended by org.apache.solr.analysis.BufferedTokenStream
                  extended by org.apache.solr.analysis.RemoveDuplicatesTokenFilter

public class RemoveDuplicatesTokenFilter
extends BufferedTokenStream

A TokenFilter which filters out Tokens at the same position and Term text as the previous token in the stream.


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
 
Field Summary
 
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
 
Constructor Summary
RemoveDuplicatesTokenFilter(TokenStream input)
           
 
Method Summary
protected  Token process(Token t)
          Process a token.
 
Methods inherited from class org.apache.solr.analysis.BufferedTokenStream
next, output, peek, pushBack, read, reset, write
 
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
 
Methods inherited from class org.apache.lucene.analysis.TokenStream
getOnlyUseNewAPI, incrementToken, next, setOnlyUseNewAPI
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

RemoveDuplicatesTokenFilter

public RemoveDuplicatesTokenFilter(TokenStream input)
Method Detail

process

protected Token process(Token t)
                 throws IOException
Description copied from class: BufferedTokenStream
Process a token. Subclasses may read more tokens from the input stream, write more tokens to the output stream, or simply return the next token to be output. Subclasses may return null if the token is to be dropped. If a subclass writes tokens to the output stream and returns a non-null Token, the returned Token is considered to be at the head of the token output stream.

Specified by:
process in class BufferedTokenStream
Throws:
IOException


Copyright © 2011 Apache Software Foundation. All Rights Reserved.