antlr

Class TokenBuffer


public class TokenBuffer
extends java.lang.Object

A Stream of Token objects fed to the parser from a Tokenizer that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

See Also:
Token, antlr.Tokenizer, TokenQueue

Field Summary

protected TokenStream
input
(package private) int
markerOffset
(package private) int
nMarkers
(package private) int
numToConsume
(package private) TokenQueue
queue

Constructor Summary

TokenBuffer(TokenStream input_)
Create a token buffer

Method Summary

int
LA(int i)
Get a lookahead token value
Token
LT(int i)
Get a lookahead token
void
consume()
Mark another token for deferred consumption
TokenStream
getInput()
return the Tokenizer (needed by ParseView)
int
mark()
Return an integer marker that can be used to rewind the buffer to its current state.
void
reset()
Reset the input buffer to empty state
void
rewind(int mark)
Rewind the token buffer to a marker.

Field Details

input

protected TokenStream input

markerOffset

(package private)  int markerOffset

nMarkers

(package private)  int nMarkers

numToConsume

(package private)  int numToConsume

queue

(package private)  TokenQueue queue

Constructor Details

TokenBuffer

public TokenBuffer(TokenStream input_)
Create a token buffer

Method Details

LA

public final int LA(int i)
            throws TokenStreamException
Get a lookahead token value

LT

public final Token LT(int i)
            throws TokenStreamException
Get a lookahead token

consume

public final void consume()
Mark another token for deferred consumption

getInput

public TokenStream getInput()
return the Tokenizer (needed by ParseView)

mark

public final int mark()
Return an integer marker that can be used to rewind the buffer to its current state.

reset

public final void reset()
Reset the input buffer to empty state

rewind

public final void rewind(int mark)
Rewind the token buffer to a marker.
Parameters:
mark - Marker returned previously from mark()