edu.jmu.decaf

Class DecafLexer

java.lang.Object

edu.jmu.decaf.DecafLexer

Direct Known Subclasses:

MyDecafLexer

public abstract class DecafLexer
extends Object

"Stub" Decaf lexer. Contains some utility functions for manipulating StringBuffer objects using Pattern regular expressions.

Constructor Summary

Constructors

Constructor and Description
DecafLexer() Generic constructor.

Method Summary

Modifier and Type	Method and Description
void	addIgnoredPattern(String regex) Add a regular expression that describes an ignored token.
void	addTokenPattern(Token.Type type, String regex) Add a regular expression that describes a particular type of token.
void	discardIgnored(StringBuffer text) Discard any ignored text.
boolean	extract(Pattern pattern, StringBuffer text) Test the given regular expression pattern against the beginning of the string buffer.
String	getLastMatch() Return the most recent text match extracted using extract(java.util.regex.Pattern, java.lang.StringBuffer).
Queue<Token>	lex(BufferedReader input, String filename) Perform lexical analysis, converting Decaf source code into a stream of Token objects.
Queue<Token>	lex(File file) Perform lexical analysis, converting Decaf source code into a stream of Token objects.
Queue<Token>	lex(InputStream input) Perform lexical analysis, converting Decaf source code into a stream of Token objects.
Queue<Token>	lex(String text) Perform lexical analysis, converting Decaf source code into a stream of Token objects.
Token	nextToken(StringBuffer text) Extract a token from the given buffer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DecafLexer

public DecafLexer()

Generic constructor. Initializes data structures.

Method Detail

lex

public Queue<Token> lex(String text)
                 throws IOException,
                        InvalidTokenException

Perform lexical analysis, converting Decaf source code into a stream of Token objects.

Parameters:

text - Input text string

Returns:

Queue of lexed tokens

Throws:

IOException - Thrown if there is an error reading from the input

InvalidTokenException - Thrown if an invalid token is encountered

lex

public Queue<Token> lex(File file)
                 throws IOException,
                        InvalidTokenException

Perform lexical analysis, converting Decaf source code into a stream of Token objects.

Parameters:

file - Input text file

Returns:

Queue of lexed tokens

Throws:

IOException - Thrown if there is an error reading from the input

InvalidTokenException - Thrown if an invalid token is encountered

lex

public Queue<Token> lex(InputStream input)
                 throws IOException,
                        InvalidTokenException

Perform lexical analysis, converting Decaf source code into a stream of Token objects.

Parameters:

input - Input text stream

Returns:

Queue of lexed tokens

Throws:

IOException - Thrown if there is an error reading from the input

InvalidTokenException - Thrown if an invalid token is encountered

lex

public Queue<Token> lex(BufferedReader input,
                        String filename)
                 throws IOException,
                        InvalidTokenException

Perform lexical analysis, converting Decaf source code into a stream of Token objects. This is the form of the method that should be overridden in actual implementations; the other forms of this method just call this one.

Parameters:

input - Input text reader

filename - Source file name (for source info)

Returns:

Queue of lexed tokens

Throws:

IOException - Thrown if there is an error reading from the input

InvalidTokenException - Thrown if an invalid token is encountered

extract

public boolean extract(Pattern pattern,
                       StringBuffer text)

Test the given regular expression pattern against the beginning of the string buffer. If the regex matches, the matching portion is deleted from the buffer and saved for later retrieval via the getLastMatch() method. Note that the matching is performed greedily.

Parameters:

pattern - Regular expression

text - Target text

Returns:

A boolean indicating whether the pattern was matched

getLastMatch

public String getLastMatch()

Return the most recent text match extracted using extract(java.util.regex.Pattern, java.lang.StringBuffer).

Returns:

Last match

addIgnoredPattern

public void addIgnoredPattern(String regex)

Add a regular expression that describes an ignored token. Any text matching this regex at the beginning will be discarded when discardIgnored(java.lang.StringBuffer) is called.

Parameters:

regex - Regular expression string

discardIgnored

public void discardIgnored(StringBuffer text)

Discard any ignored text. Check all regexes registered with addIgnoredPattern(java.lang.String) one at a time, discarding any text matched.

Parameters:

text - Target text

addTokenPattern

public void addTokenPattern(Token.Type type,
                            String regex)

Add a regular expression that describes a particular type of token. These patterns are used by nextToken(java.lang.StringBuffer) to extract tokens from the beginning of a StringBuffer. The order in which this function is called is significant; patterns that are registered earlier will be matched first.

Parameters:

type - Token type

regex - Regular expression string

nextToken

public Token nextToken(StringBuffer text)

Extract a token from the given buffer. Checks all regexes registered with addTokenPattern(edu.jmu.decaf.Token.Type, java.lang.String) one at a time, returning the next token matched. The patterns are matched in the order that they were added.

Parameters:

text - Target text

Returns:

The new token, or null if there is no valid token