lexer: skip interpreter line in any source buffer

Skip interpreter lines in any source buffer and handle the skipping in the
lexer itself, to avoid reporting wrongly shifted token offsets to the
compiler, resulting in wrong error locations and source contexts.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>

1 files changed, 21 insertions, 0 deletions

diff --git a/tests/custom/03_bugs/25_lexer_shifted_offsets b/tests/custom/03_bugs/25_lexer_shifted_offsets
new file mode 100644
index 0000000..db10121
--- /dev/null
+++ b/tests/custom/03_bugs/25_lexer_shifted_offsets
@@ -0,0 +1,21 @@
+When lexing a source buffer with a non-zero offset, e.g. due to a
+skipped interpreter line, lexical tokens reported a wrong offset
+to the compiler, causing error locations and source context lines
+to be incorrectly shifted.
+
+-- Testcase --
+#!/usr/bin/env ucode
+{%
+	die("Error");
+%}
+-- End --
+
+-- Expect stderr --
+Error
+In line 3, byte 12:
+
+ `    die("Error");`
+  Near here -----^
+
+
+-- End --