Decaf Compiler
Classes | Typedefs | Enumerations | Functions
token.h File Reference

Tokens and regular expressions. More...

#include "common.h"

Go to the source code of this file.

Classes

struct  Token
 Single token. More...
 
struct  TokenQueue
 Linked list of tokens. More...
 

Typedefs

typedef regex_t Regex
 Compiled regular expression. More...
 
typedef enum TokenType TokenType
 Valid token types. More...
 
typedef struct Token Token
 Single token. More...
 
typedef struct TokenQueue TokenQueue
 Linked list of tokens. More...
 

Enumerations

enum  TokenType {
  ID , DECLIT , HEXLIT , STRLIT ,
  KEY , SYM
}
 Valid token types. More...
 

Functions

RegexRegex_new (const char *regex)
 Allocate and compile a new regular expression. More...
 
bool Regex_match (Regex *regex, const char *text, char *match)
 Match a regular expression against some text. More...
 
void Regex_free (Regex *regex)
 Deallocate a regular expression. More...
 
const char * TokenType_to_string (TokenType type)
 Convert a token type to a string for output. More...
 
bool token_str_eq (const char *str1, const char *str2)
 Check string equality for tokens. Limits comparison to MAX_TOKEN_LEN for safety. More...
 
TokenToken_new (TokenType type, const char *text, int line)
 Allocate and initialize a new token. More...
 
void Token_free (Token *token)
 Deallocate a token. More...
 
TokenQueueTokenQueue_new ()
 Allocate and initialize a new, empty queue of tokens. More...
 
void TokenQueue_add (TokenQueue *queue, Token *token)
 Add a token to a queue. More...
 
TokenTokenQueue_peek (TokenQueue *queue)
 Return the next token from a queue without removing it (first-in-first-out) More...
 
TokenTokenQueue_remove (TokenQueue *queue)
 Remove a token from a queue (first-in-first-out) More...
 
bool TokenQueue_is_empty (TokenQueue *queue)
 Check whether a queue is empty. More...
 
size_t TokenQueue_size (TokenQueue *queue)
 Calculate size of the queue. More...
 
void TokenQueue_print (TokenQueue *queue, FILE *out)
 Print a queue to the given file descriptor (debug output) More...
 
void TokenQueue_free (TokenQueue *queue)
 Deallocate a token queue. More...
 

Detailed Description

Tokens and regular expressions.

The structures and functions declared in this file will likely only be useful in Project 1 (lexing).

Typedef Documentation

◆ Regex

typedef regex_t Regex

Compiled regular expression.

This is just a thin wrapper around POSIX regular expressions, so Regex is just a typedef for regex_t.

Allocate with Regex_new and de-allocate with Regex_free.

Methods:

◆ Token

typedef struct Token Token

Single token.

Allocate with Token_new and de-allocate with Token_free.

◆ TokenQueue

typedef struct TokenQueue TokenQueue

Linked list of tokens.

Allocate with TokenQueue_new and de-allocate with TokenQueue_free.

Methods:

◆ TokenType

typedef enum TokenType TokenType

Valid token types.

May be any of the following:

  • ID - identifier
  • DECLIT - positive decimal integer literal
  • HEXLIT - positive hexadecimal integer literal (prefixed by '0x')
  • STRLIT - string literal (enclosed in quote marks)
  • KEY - keyword
  • SYM - symbol (including multi-character symbols)

Enumeration Type Documentation

◆ TokenType

enum TokenType

Valid token types.

May be any of the following:

  • ID - identifier
  • DECLIT - positive decimal integer literal
  • HEXLIT - positive hexadecimal integer literal (prefixed by '0x')
  • STRLIT - string literal (enclosed in quote marks)
  • KEY - keyword
  • SYM - symbol (including multi-character symbols)

Function Documentation

◆ Regex_free()

void Regex_free ( Regex regex)

Deallocate a regular expression.

Parameters
regexCompiled regular expression to deallocate

◆ Regex_match()

bool Regex_match ( Regex regex,
const char *  text,
char *  match 
)

Match a regular expression against some text.

If the regex matches, the matched text will be written into the given match buffer.

Parameters
regexCompiled regular expression to match against
textText to match
matchCharacter buffer (must be at least MAX_TOKEN_LEN long)
Returns
True if and only if the text matched the regular expression

◆ Regex_new()

Regex* Regex_new ( const char *  regex)

Allocate and compile a new regular expression.

Parameters
regexString containing regular expression to compile
Returns
Newly-compiled regular expression

◆ Token_free()

void Token_free ( Token token)

Deallocate a token.

Parameters
tokenToken to deallocate

◆ Token_new()

Token* Token_new ( TokenType  type,
const char *  text,
int  line 
)

Allocate and initialize a new token.

Make sure Token_free() is called to deallocate the token, otherwise there will be a memory leak.

Parameters
typeType of new token
textRaw text for new token
lineLine number of new token
Returns
Newly-created token

◆ token_str_eq()

bool token_str_eq ( const char *  str1,
const char *  str2 
)

Check string equality for tokens. Limits comparison to MAX_TOKEN_LEN for safety.

Parameters
str1First string to compare
str2Second string to compare
Returns
True if strings are equal; false otherwise

◆ TokenQueue_add()

void TokenQueue_add ( TokenQueue queue,
Token token 
)

Add a token to a queue.

Parameters
queueQueue to add to
tokenToken to add

◆ TokenQueue_free()

void TokenQueue_free ( TokenQueue queue)

Deallocate a token queue.

Also deallocates any remaining tokens

Parameters
queueQueue to deallocate

◆ TokenQueue_is_empty()

bool TokenQueue_is_empty ( TokenQueue queue)

Check whether a queue is empty.

Parameters
queueQueue to check
Returns
True if and only if the queue is empty

◆ TokenQueue_new()

TokenQueue* TokenQueue_new ( )

Allocate and initialize a new, empty queue of tokens.

Returns
Newly-created queue of tokens

◆ TokenQueue_peek()

Token* TokenQueue_peek ( TokenQueue queue)

Return the next token from a queue without removing it (first-in-first-out)

Parameters
queueQueue to look at
Returns
Token extracted

◆ TokenQueue_print()

void TokenQueue_print ( TokenQueue queue,
FILE *  out 
)

Print a queue to the given file descriptor (debug output)

Parameters
queueQueue to print
outFile stream to print to

◆ TokenQueue_remove()

Token* TokenQueue_remove ( TokenQueue queue)

Remove a token from a queue (first-in-first-out)

Parameters
queueQueue to remove from
Returns
Token removed

◆ TokenQueue_size()

size_t TokenQueue_size ( TokenQueue queue)

Calculate size of the queue.

Parameters
queueQueue to check
Returns
Number of tokens in the queue

◆ TokenType_to_string()

const char* TokenType_to_string ( TokenType  type)

Convert a token type to a string for output.

Parameters
typeType to convert
Returns
Static const string representation of the given type