token.h File Reference

Tokens and regular expressions. More...

#include "common.h"

Go to the source code of this file.

Classes
struct	Token
	Single token. More...
struct	TokenQueue
	Linked list of tokens. More...

Typedefs
typedef regex_t	Regex
	Compiled regular expression. More...
typedef enum TokenType	TokenType
	Valid token types. More...
typedef struct Token	Token
	Single token. More...
typedef struct TokenQueue	TokenQueue
	Linked list of tokens. More...

Enumerations
enum	TokenType {   ID , DECLIT , HEXLIT , STRLIT ,   KEY , SYM }
	Valid token types. More...

Functions
Regex *	Regex_new (const char *regex)
	Allocate and compile a new regular expression. More...
bool	Regex_match (Regex *regex, const char *text, char *match)
	Match a regular expression against some text. More...
void	Regex_free (Regex *regex)
	Deallocate a regular expression. More...
const char *	TokenType_to_string (TokenType type)
	Convert a token type to a string for output. More...
bool	token_str_eq (const char *str1, const char *str2)
	Check string equality for tokens. Limits comparison to MAX_TOKEN_LEN for safety. More...
Token *	Token_new (TokenType type, const char *text, int line)
	Allocate and initialize a new token. More...
void	Token_free (Token *token)
	Deallocate a token. More...
TokenQueue *	TokenQueue_new ()
	Allocate and initialize a new, empty queue of tokens. More...
void	TokenQueue_add (TokenQueue *queue, Token *token)
	Add a token to a queue. More...
Token *	TokenQueue_peek (TokenQueue *queue)
	Return the next token from a queue without removing it (first-in-first-out) More...
Token *	TokenQueue_remove (TokenQueue *queue)
	Remove a token from a queue (first-in-first-out) More...
bool	TokenQueue_is_empty (TokenQueue *queue)
	Check whether a queue is empty. More...
size_t	TokenQueue_size (TokenQueue *queue)
	Calculate size of the queue. More...
void	TokenQueue_print (TokenQueue *queue, FILE *out)
	Print a queue to the given file descriptor (debug output) More...
void	TokenQueue_free (TokenQueue *queue)
	Deallocate a token queue. More...

Detailed Description

Tokens and regular expressions.

The structures and functions declared in this file will likely only be useful in Project 1 (lexing).

Typedef Documentation

Regex

typedef regex_t Regex

Compiled regular expression.

This is just a thin wrapper around POSIX regular expressions, so Regex is just a typedef for regex_t.

Allocate with Regex_new and de-allocate with Regex_free.

Methods:

• Regex_match

Token

typedef struct Token Token

Single token.

Allocate with Token_new and de-allocate with Token_free.

TokenQueue

typedef struct TokenQueue TokenQueue

Linked list of tokens.

Allocate with TokenQueue_new and de-allocate with TokenQueue_free.

Methods:

• TokenQueue_peek
• TokenQueue_remove
• TokenQueue_is_empty
• TokenQueue_size
• TokenQueue_print

TokenType

typedef enum TokenType TokenType

Valid token types.

May be any of the following:

• ID - identifier
• DECLIT - positive decimal integer literal
• HEXLIT - positive hexadecimal integer literal (prefixed by '0x')
• STRLIT - string literal (enclosed in quote marks)
• KEY - keyword
• SYM - symbol (including multi-character symbols)

Enumeration Type Documentation

TokenType

enum TokenType

Valid token types.

May be any of the following:

• ID - identifier
• DECLIT - positive decimal integer literal
• HEXLIT - positive hexadecimal integer literal (prefixed by '0x')
• STRLIT - string literal (enclosed in quote marks)
• KEY - keyword
• SYM - symbol (including multi-character symbols)

Function Documentation

Regex_free()

void Regex_free

(

Regex *

regex

)

Deallocate a regular expression.

Parameters

regex

Compiled regular expression to deallocate

Regex_match()

bool Regex_match	(	Regex *	regex,
		const char *	text,
		char *	match
	)

Match a regular expression against some text.

If the regex matches, the matched text will be written into the given match buffer.

Parameters

regex	Compiled regular expression to match against
text	Text to match
match	Character buffer (must be at least MAX_TOKEN_LEN long)

Returns

True if and only if the text matched the regular expression

Regex_new()

Regex* Regex_new

(

const char *

regex

)

Allocate and compile a new regular expression.

Parameters

regex

String containing regular expression to compile

Returns

Newly-compiled regular expression

Token_free()

void Token_free

(

Token *

token

)

Deallocate a token.

Parameters

token

Token to deallocate

Token_new()

Token* Token_new	(	TokenType	type,
		const char *	text,
		int	line
	)

Allocate and initialize a new token.

Make sure Token_free() is called to deallocate the token, otherwise there will be a memory leak.

Parameters

type	Type of new token
text	Raw text for new token
line	Line number of new token

Returns

Newly-created token

token_str_eq()

bool token_str_eq	(	const char *	str1,
		const char *	str2
	)

Check string equality for tokens. Limits comparison to MAX_TOKEN_LEN for safety.

Parameters

str1	First string to compare
str2	Second string to compare

Returns

True if strings are equal; false otherwise

TokenQueue_add()

void TokenQueue_add	(	TokenQueue *	queue,
		Token *	token
	)

Add a token to a queue.

Parameters

queue	Queue to add to
token	Token to add

TokenQueue_free()

void TokenQueue_free

(

TokenQueue *

queue

)

Deallocate a token queue.

Also deallocates any remaining tokens

Parameters

queue

Queue to deallocate

TokenQueue_is_empty()

bool TokenQueue_is_empty

(

TokenQueue *

queue

)

Check whether a queue is empty.

Parameters

queue

Queue to check

Returns

True if and only if the queue is empty

TokenQueue_new()

TokenQueue* TokenQueue_new

(

)

Allocate and initialize a new, empty queue of tokens.

Returns

Newly-created queue of tokens

TokenQueue_peek()

Token* TokenQueue_peek

(

TokenQueue *

queue

)

Return the next token from a queue without removing it (first-in-first-out)

Parameters

queue

Queue to look at

Returns

Token extracted

TokenQueue_print()

void TokenQueue_print	(	TokenQueue *	queue,
		FILE *	out
	)

Print a queue to the given file descriptor (debug output)

Parameters

queue	Queue to print
out	File stream to print to

TokenQueue_remove()

Token* TokenQueue_remove

(

TokenQueue *

queue

)

Remove a token from a queue (first-in-first-out)

Parameters

queue

Queue to remove from

Returns

Token removed

TokenQueue_size()

size_t TokenQueue_size

(

TokenQueue *

queue

)

Calculate size of the queue.

Parameters

queue

Queue to check

Returns

Number of tokens in the queue

TokenType_to_string()

const char* TokenType_to_string

(

TokenType

type

)

Convert a token type to a string for output.

Parameters

type	Type to convert

Returns

Static const string representation of the given type