Decaf Compiler
|
Tokens and regular expressions. More...
#include "common.h"
Go to the source code of this file.
Classes | |
struct | Token |
Single token. More... | |
struct | TokenQueue |
Linked list of tokens. More... | |
Typedefs | |
typedef regex_t | Regex |
Compiled regular expression. | |
typedef enum TokenType | TokenType |
Valid token types. | |
typedef struct Token | Token |
Single token. | |
typedef struct TokenQueue | TokenQueue |
Linked list of tokens. | |
Enumerations | |
enum | TokenType { ID , DECLIT , HEXLIT , STRLIT , KEY , SYM } |
Valid token types. More... | |
Functions | |
Regex * | Regex_new (const char *regex) |
Allocate and compile a new regular expression. | |
bool | Regex_match (Regex *regex, const char *text, char *match) |
Match a regular expression against some text. | |
void | Regex_free (Regex *regex) |
Deallocate a regular expression. | |
const char * | TokenType_to_string (TokenType type) |
Convert a token type to a string for output. | |
bool | token_str_eq (const char *str1, const char *str2) |
Check string equality for tokens. Limits comparison to MAX_TOKEN_LEN for safety. | |
Token * | Token_new (TokenType type, const char *text, int line) |
Allocate and initialize a new token. | |
void | Token_free (Token *token) |
Deallocate a token. | |
TokenQueue * | TokenQueue_new (void) |
Allocate and initialize a new, empty queue of tokens. | |
void | TokenQueue_add (TokenQueue *queue, Token *token) |
Add a token to a queue. | |
Token * | TokenQueue_peek (TokenQueue *queue) |
Return the next token from a queue without removing it (first-in-first-out) | |
Token * | TokenQueue_remove (TokenQueue *queue) |
Remove a token from a queue (first-in-first-out) | |
bool | TokenQueue_is_empty (TokenQueue *queue) |
Check whether a queue is empty. | |
size_t | TokenQueue_size (TokenQueue *queue) |
Calculate size of the queue. | |
void | TokenQueue_print (TokenQueue *queue, FILE *out) |
Print a queue to the given file descriptor (debug output) | |
void | TokenQueue_free (TokenQueue *queue) |
Deallocate a token queue. | |
Tokens and regular expressions.
The structures and functions declared in this file will likely only be useful in Project 1 (lexing).
typedef regex_t Regex |
Compiled regular expression.
This is just a thin wrapper around POSIX regular expressions, so Regex
is just a typedef for regex_t
.
Allocate with Regex_new and de-allocate with Regex_free.
Methods:
Single token.
Allocate with Token_new and de-allocate with Token_free.
typedef struct TokenQueue TokenQueue |
Valid token types.
May be any of the following:
ID
- identifier DECLIT
- positive decimal integer literal HEXLIT
- positive hexadecimal integer literal (prefixed by '0x') STRLIT
- string literal (enclosed in quote marks) KEY
- keyword SYM
- symbol (including multi-character symbols) enum TokenType |
Valid token types.
May be any of the following:
ID
- identifier DECLIT
- positive decimal integer literal HEXLIT
- positive hexadecimal integer literal (prefixed by '0x') STRLIT
- string literal (enclosed in quote marks) KEY
- keyword SYM
- symbol (including multi-character symbols) void Regex_free | ( | Regex * | regex | ) |
Deallocate a regular expression.
regex | Compiled regular expression to deallocate |
bool Regex_match | ( | Regex * | regex, |
const char * | text, | ||
char * | match | ||
) |
Match a regular expression against some text.
If the regex matches, the matched text will be written into the given match buffer.
regex | Compiled regular expression to match against |
text | Text to match |
match | Character buffer (must be at least MAX_TOKEN_LEN long) |
Regex * Regex_new | ( | const char * | regex | ) |
Allocate and compile a new regular expression.
regex | String containing regular expression to compile |
Allocate and initialize a new token.
Make sure Token_free() is called to deallocate the token, otherwise there will be a memory leak.
type | Type of new token |
text | Raw text for new token |
line | Line number of new token |
bool token_str_eq | ( | const char * | str1, |
const char * | str2 | ||
) |
Check string equality for tokens. Limits comparison to MAX_TOKEN_LEN
for safety.
str1 | First string to compare |
str2 | Second string to compare |
void TokenQueue_add | ( | TokenQueue * | queue, |
Token * | token | ||
) |
Add a token to a queue.
queue | Queue to add to |
token | Token to add |
void TokenQueue_free | ( | TokenQueue * | queue | ) |
Deallocate a token queue.
Also deallocates any remaining tokens
queue | Queue to deallocate |
bool TokenQueue_is_empty | ( | TokenQueue * | queue | ) |
Check whether a queue is empty.
queue | Queue to check |
TokenQueue * TokenQueue_new | ( | void | ) |
Allocate and initialize a new, empty queue of tokens.
Token * TokenQueue_peek | ( | TokenQueue * | queue | ) |
Return the next token from a queue without removing it (first-in-first-out)
queue | Queue to look at |
void TokenQueue_print | ( | TokenQueue * | queue, |
FILE * | out | ||
) |
Print a queue to the given file descriptor (debug output)
queue | Queue to print |
out | File stream to print to |
Token * TokenQueue_remove | ( | TokenQueue * | queue | ) |
Remove a token from a queue (first-in-first-out)
queue | Queue to remove from |
size_t TokenQueue_size | ( | TokenQueue * | queue | ) |
Calculate size of the queue.
queue | Queue to check |
const char * TokenType_to_string | ( | TokenType | type | ) |
Convert a token type to a string for output.
type | Type to convert |