org.apache.lucene.analysis.standard

Class StandardTokenizer

Implemented Interfaces:
StandardTokenizerConstants

public class StandardTokenizer
extends Tokenizer
implements StandardTokenizerConstants

A grammar-based tokenizer constructed with JavaCC.

This should be a good tokenizer for most European-language documents.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

Field Summary

Token
jj_nt
Token
token
StandardTokenizerTokenManager
token_source

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from interface org.apache.lucene.analysis.standard.StandardTokenizerConstants

ACRONYM, ALPHA, ALPHANUM, APOSTROPHE, CJK, COMPANY, DEFAULT, DIGIT, EMAIL, EOF, HAS_DIGIT, HOST, LETTER, NOISE, NUM, P, tokenImage

Constructor Summary

StandardTokenizer(Reader reader)
Constructs a tokenizer for this Reader.
StandardTokenizer(CharStream stream)
StandardTokenizer(StandardTokenizerTokenManager tm)

Method Summary

void
ReInit(CharStream stream)
void
ReInit(StandardTokenizerTokenManager tm)
void
disable_tracing()
void
enable_tracing()
ParseException
generateParseException()
Token
getNextToken()
Token
getToken(int index)
Token
next()
Returns the next token in the stream, or null at EOS.

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close

Methods inherited from class org.apache.lucene.analysis.TokenStream

close, next

Field Details

jj_nt

public Token jj_nt

token

public Token token

token_source

public StandardTokenizerTokenManager token_source

Constructor Details

StandardTokenizer

public StandardTokenizer(Reader reader)
Constructs a tokenizer for this Reader.

StandardTokenizer

public StandardTokenizer(CharStream stream)

StandardTokenizer

public StandardTokenizer(StandardTokenizerTokenManager tm)

Method Details

ReInit

public void ReInit(CharStream stream)

ReInit

public void ReInit(StandardTokenizerTokenManager tm)

disable_tracing

public final void disable_tracing()

enable_tracing

public final void enable_tracing()

generateParseException

public ParseException generateParseException()

getNextToken

public final Token getNextToken()

getToken

public final Token getToken(int index)

next

public final Token next()
            throws ParseException,
                   IOException
Returns the next token in the stream, or null at EOS.

The returned token's type is set to an element of StandardTokenizerConstants.tokenImage.

Overrides:
next in interface TokenStream

Copyright © 2000-2007 Apache Software Foundation. All Rights Reserved.