diff options
| author | Jo-Philipp Wich <jo@mein.io> | 2024-09-23 23:33:47 +0200 |
|---|---|---|
| committer | Jo-Philipp Wich <jo@mein.io> | 2024-09-23 23:33:47 +0200 |
| commit | 855854f6c2ae2e667f6bbfd5f67caab32b4ebf86 (patch) | |
| tree | 11d99de43b268fa1b43d278299eb5f2f4ced8963 /include | |
| parent | 328a50ff82c9bf089dcd381d404dece683ef54d2 (diff) | |
lexer: emit comment and template statement block tokens
Tweak the token stream reported by the lexer in order to make it more useful
for alternative, non-compilation downstream parse processes such as code
intelligence gathering within a language server implementation.
- Instead of silently discarding source code comments in the lexing phase,
emit TK_COMMENT tokens which is useful to e.g. parse type annotations and
other structured information.
- Do not silently discard TK_LSTM tokens but report them to downstream
parsers instead.
- Do not silently emit TK_RSTM tokens as TK_SCOL but report them as-is to
downstrem parsers.
- Adjust the byte code compiler to properly deal with the changed token
reporting by discarding incoming TK_COMMENT and TK_LSTM tokens and by
remapping read TK_RSTM tokens to the TK_SCOL type.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Diffstat (limited to 'include')
| -rw-r--r-- | include/ucode/lexer.h | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/include/ucode/lexer.h b/include/ucode/lexer.h index 8929731..8dcba7a 100644 --- a/include/ucode/lexer.h +++ b/include/ucode/lexer.h @@ -121,12 +121,14 @@ typedef enum { TK_EXPORT, TK_EOF, + TK_COMMENT, TK_ERROR } uc_tokentype_t; typedef enum { UC_LEX_IDENTIFY_BLOCK, UC_LEX_BLOCK_EXPRESSION_EMIT_TAG, + UC_LEX_BLOCK_STATEMENT_EMIT_TAG, UC_LEX_BLOCK_COMMENT, UC_LEX_IDENTIFY_TOKEN, UC_LEX_PLACEHOLDER_START, |