|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.standard.StandardTokenizer
public class StandardTokenizer
A grammar-based tokenizer constructed with JFlex
This should be a good tokenizer for most European-language documents:
Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.
Field Summary |
---|
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
---|
input |
Constructor Summary | |
---|---|
StandardTokenizer(Reader input)
Creates a new instance of the StandardTokenizer . |
|
StandardTokenizer(Reader input,
boolean replaceInvalidAcronym)
Creates a new instance of the StandardTokenizer . |
Method Summary | |
---|---|
int |
getMaxTokenLength()
|
boolean |
isReplaceInvalidAcronym()
Deprecated. Remove in 3.X and make true the only valid value |
Token |
next(Token result)
Returns the next token in the stream, or null at EOS. |
void |
reset()
Resets this stream to the beginning. |
void |
reset(Reader reader)
Expert: Reset the tokenizer to a new reader. |
void |
setMaxTokenLength(int length)
Set the max allowed token length. |
void |
setReplaceInvalidAcronym(boolean replaceInvalidAcronym)
Deprecated. Remove in 3.X and make true the only valid value See https://issues.apache.org/jira/browse/LUCENE-1068 |
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
---|
close |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
---|
next |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public StandardTokenizer(Reader input)
StandardTokenizer
. Attaches the
input
to a newly created JFlex scanner.
public StandardTokenizer(Reader input, boolean replaceInvalidAcronym)
StandardTokenizer
. Attaches
the input
to the newly created JFlex scanner.
input
- The input readerreplaceInvalidAcronym
- Set to true to replace mischaracterized acronyms with HOST.
See http://issues.apache.org/jira/browse/LUCENE-1068Method Detail |
---|
public void setMaxTokenLength(int length)
public int getMaxTokenLength()
setMaxTokenLength(int)
public Token next(Token result) throws IOException
TokenStream
This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):
Token.clear()
before setting the fields in it & returning itTokenFilter
is considered a consumer.
next
in class TokenStream
result
- a Token that may or may not be used to return
IOException
public void reset() throws IOException
TokenStream
reset
in class TokenStream
IOException
public void reset(Reader reader) throws IOException
Tokenizer
reset
in class Tokenizer
IOException
public boolean isReplaceInvalidAcronym()
public void setReplaceInvalidAcronym(boolean replaceInvalidAcronym)
replaceInvalidAcronym
- Set to true to replace mischaracterized acronyms as HOST.
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |