Merge pull request #164 from jow-/docs-improvements

docs: various improvements

4 files changed, 686 insertions, 3 deletions

diff --git a/docs/CNAME b/docs/CNAME
new file mode 100644
index 0000000..deb43a3
--- /dev/null
+++ b/docs/CNAME
@@ -0,0 +1 @@
+ucode.mein.io
diff --git a/docs/README.md b/docs/README.md
index 17fffbe..660f175 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,5 +1,659 @@
-# ucode Documentation
+# The ucode Scripting Language
 
-## Module Reference
+The ucode language is a tiny general purpose scripting language featuring a
+syntax closely resembling ECMAScript. It can be used in a stand-alone manner
+by using the ucode command line interpreter or embedded into host applications
+by linking libucode and utilizing its C language API. Additionally, ucode can
+be invoked in template mode where control flow and expression logic statements
+are embedded in Jinja-like markup blocks.
 
- - [ucode module documentation](index.html)
+Besides aiming for small size, the major design goals of ucode are the ability
+to trivially read and write JSON data, good embeddability into C applications,
+template capabilities for output formatting, extensiblity through loadable
+native extension modules and a straightforward set of built-in functions
+mimicking those found in the Perl 5 language.
+
+## History and Motivation
+
+In spring 2021 it has been decided to rewrite the OpenWrt firewall framework on
+top of nftables with the goal to replace the then current C application with a
+kind of preprocessor generating nftables rulesets using a set of templates
+instead of relying on built-in hardcoded rules like its predecessor.
+
+That decision spurred the development of *ucode*, initially meant to be a
+simple template processor solely for the OpenWrt nftables firewall but quickly
+evolving into a general purpose scripting language suitable for a wider range
+of system scripting tasks.
+
+Despite OpenWrt predominantly relying on POSIX shell and Lua as system
+scripting languages already, a new solution was needed to accomodate the needs
+of the new firewall implementation; mainly the ability to efficiently deal with
+JSON data and complex data structures such as arrays and dictionaries and the
+ability to closely interface with OpenWrt's *ubus* message bus system.
+
+Throughout the design process of the new firewall and its template processor,
+the following design goals were defined for the *ucode* scripting language:
+
+ - Ability to embed code logic fragments such as control flow statements,
+   function calls or arithmetic expressions into plain text templates, using
+   a block syntax and functionality roughly inspired by Jinja templates
+ - Built-in support for JSON data parsing and serialization, without the need
+   for external libraries
+ - Distinct array and object types (compared to Lua's single table datatype)
+ - Distinct integer and float types and guaranteed 64bit integer range
+ - Built-in support for bit operations
+ - Built-in support for (POSIX) regular expressions
+ - A comprehensive set of built-in standard functions, inspired by the core
+   functions found in the Perl 5 interpreter
+ - Staying as close to ECMAScript syntax as possible due to higher developer
+   familiarity and to be able to reuse existing tooling such as editor syntax
+   highlighting
+ - Bindings for all relevant Linux and OpenWrt APIs, such as *ubus*, *uci*,
+   *uloop*, *netlink* etc.
+ - Procedural, synchronous programming flow
+ - Very small executable size (the interpreter and runtime is currently around
+   64KB on ARM Cortex A9)
+ - Embeddability into C host applications
+
+Summarized, *ucode* can be described as synchronous ECMAScript without the
+object oriented standard library.
+
+
+## Installation
+
+### OpenWrt
+
+In OpenWrt 22.03 and later, *ucode* should already be preinstalled. If not,
+it can be installed via the package manager, using the `opkg install ucode`
+command.
+
+### MacOS
+
+To build on MacOS, first install *cmake* and *json-c* via
+[Homebrew](https://brew.sh/), then clone the ucode repository and execute
+*cmake* followed by *make*:
+
+    $ brew install cmake json-c
+    $ git clone https://github.com/jow-/ucode.git
+    $ cd ucode/
+    $ cmake -DUBUS_SUPPORT=OFF -DUCI_SUPPORT=OFF -DULOOP_SUPPORT=OFF .
+    $ make
+    $ sudo make install
+
+### Debian
+
+The ucode repository contains build recipes for Debian packages, to build .deb
+packages for local installation, first install required development packages,
+then clone the repository and invoke *dpkg-buildpackage* to produce the binary
+package files:
+
+    $ sudo apt-get install build-essential devscripts debhelper libjson-c-dev cmake pkg-config
+    $ git clone https://github.com/jow-/ucode.git
+    $ cd ucode/
+    $ dpkg-buildpackage -b -us -uc
+    $ sudo dpkg -i ../ucode*.deb ../libucode*.deb
+
+### Other Linux systems
+
+To install ucode from source on other systems, ensure that the json-c library
+and associated development headers are installed, then clone and compile the
+ucode repository:
+
+    $ git clone https://github.com/jow-/ucode.git
+    $ cd ucode/
+    $ cmake -DUBUS_SUPPORT=OFF -DUCI_SUPPORT=OFF -DULOOP_SUPPORT=OFF .
+    $ make
+    $ sudo make install
+
+
+## Syntax
+
+### Template mode
+
+By default, *ucode* is executed in *raw mode*, means it expects a given source
+file to only contain script code. By invoking the ucode interpreter with the
+`-T` flag or by using the `utpl` alias, the *ucode* interpreter is switched
+into *template mode* where the source file is expected to be a plaintext file
+containing *template blocks* containing ucode script expressions or comments.
+
+#### Block types
+
+There are three kinds of blocks; *expression blocks*, *statement blocks* and
+*comment blocks*. The former two embed code logic using ucode's JavaScript-like
+syntax while the latter comment block type is simply discarded during
+processing.
+
+
+##### 1. Statement block
+
+Statement blocks are enclosed in an opening `{%` and a closing `%}` tag and
+may contain any number of script code statements, even entire programs.
+
+It is allowed to omit the closing `%}` of a statement block to parse the
+entire remaining source text after the opening tag as ucode script.
+
+By default, statement blocks produce no output and the entire block is
+reduced to an empty string during template evaluation but contained script
+code might invoke functions such as `print()` to explicitly output contents.
+
+For example the following template would result in `The epoch is odd` or
+`The epoch is even`, depending on the current epoch value:
+
+`The epoch is {% if (time() % 2): %}odd{% else %}even{% endif %}!`
+
+
+##### 2. Expression block
+
+Expression blocks are enclosed in an opening `{{` and a closing `}}` tag and
+may only contain a single expression statement (multiple expressions may be
+chained with comma). The implicit result of the rightmost evaluated expression
+is used as output when processing the block.
+
+For example the template `Hello world, {{ getenv("USER") }}!` would result in
+the output "Hello world, user!" where `user` would correspond to the name of
+the current user executing the ucode interpreter.
+
+
+##### 3. Comment block
+
+Comment blocks, which are denoted with an opening `{#` and a closing `#}` tag
+may contain arbitrary text except the closing `#}` tag itself. Comments blocks
+are completely stripped during processing and are replaced with an empty string.
+
+The following example template would result in the output "Hello world":
+
+`Hello {# mad #}word`
+
+
+#### Whitespace handling
+
+Each block start tag may be suffixed with a dash to strip any whitespace
+before the block and likewise any block end tag may be prefixed with a dash
+to strip any whitespace following the block.
+
+Without using whitespace stripping, the following example:
+
+```
+This is a first line
+{% for (x in [1, 2, 3]): %}
+This is item {{ x }}.
+{% endfor %}
+This is the last line
+```
+
+Would result in the following output:
+
+```
+This is a first line
+
+This is item 1.
+This is item 2.
+This is item 3.
+
+This is the last line
+```
+
+By adding a trailing dash to apply whitespace stripping after the block, the
+empty lines can be eliminated:
+
+```
+This is a first line
+{% for (x in [1, 2, 3]): -%}
+This is item {{ x }}.
+{% endfor -%}
+This is the last line
+```
+
+Output:
+
+```
+This is a first line
+This is item 1.
+This is item 2.
+This is item 3.
+This is the last line
+```
+
+By applying whitespace stripping before the block, all lines can be joined
+into a single output line:
+
+```
+This is a first line
+{%- for (x in [1, 2, 3]): -%}
+This is item {{ x }}.
+{%- endfor -%}
+This is the last line
+```
+
+Output:
+
+```
+This is a first lineThis is item 1.This is item 2.This is item 3.This is the last line
+```
+
+### Script syntax
+
+The ucode script language - used either within statement and expression blocks
+or throughout the entire file in *raw mode*, uses untyped variables and employs
+a simplified JavaScript like syntax.
+
+The language implements function scoping and differentiates between local and
+global variables. Each function has its own private scope while executing and
+local variables declared inside a function are not accessible in the outer
+calling scope.
+
+#### 1. Data types
+
+Ucode supports seven different basic types as well as two additional special
+types; function values and ressource values. The supported types are:
+
+ - Boolean values (`true` or `false`)
+ - Integer values (`-9223372036854775808` to `+9223372036854775807`)
+ - Double values (`-1.7e308` to `+1.7e308`)
+ - String values (e.g. `'Hello world!'` or `"Sunshine \u2600!"`)
+ - Array values (e.g. `[1, false, "foo"]`)
+ - Object values (e.g. `{ foo: true, "bar": 123 }`)
+ - Null value (`null`)
+
+Ucode utilizes reference counting to manage memory used for variables and values
+and frees data automatically as soon as values go out of scope.
+
+Numeric values are either stored as signed 64bit integers or as IEEE 756 double
+value. Conversion between integer and double values can happen implicitly, e.g.
+through numeric operations, or explicitely, e.g. by invoking functions such as
+`int()`.
+
+#### 2. Variables
+
+Variable names must start with a letter or an underscore and may only contain
+the characters `A`..`Z`, `a`..`z`, `0`..`9` or `_`. By prefixing a variable
+name with the keyword `let`, it is declared in the local block scope only
+and not visible outside anymore.
+
+Variables may also be declared using the `const` keyword. Such variables follow
+the same scoping rules as `let` declared ones but they cannot be modified after
+they have been declared. Any attempt to do so will result in a syntax error
+during compilation.
+
+```javascript
+{%
+
+  a = 1;  // global variable assignment
+
+  function test() {
+    let b = 2;  // declare `b` as local variable
+    a = 2;        // overwrite global a
+  }
+
+  test();
+
+  print(a, "\n");  // outputs "2"
+  print(b, "\n");  // outputs nothing
+
+  const c = 3;
+  print(c, "\n");  // outputs "3"
+
+  c = 4;           // raises syntax error
+  c++;             // raises syntax error
+
+  const d;         // raises syntax error, const variables must
+                   // be initialized at declaration time
+
+%}
+```
+
+#### 3. Control statements
+
+Similar to JavaScript, ucode supports `if`, `for` and `while` statements to
+control execution flow.
+
+##### 3.1. Conditional statement
+
+If/else blocks can be used to execute statements depending on a condition.
+
+```javascript
+{%
+
+  user = getenv("USER");
+
+  if (user == "alice") {
+      print("Hello Alice!\n");
+  }
+  else if (user == "bob") {
+      print("Hello Bob!\n");
+  }
+  else {
+      print("Hello guest!\n");
+  }
+
+%}
+```
+
+If only a single statement is wrapped by an if or else branch, the enclosing
+curly braces may be omitted:
+
+```javascript
+{%
+
+  if (rand() == 3)
+      print("This is quite unlikely\n");
+
+%}
+```
+
+##### 3.2. Loop statements
+
+Ucode script supports three different flavors of loop control statements; a
+`while` loop that executes enclosed statements as long as the loop condition is
+fulfilled, a `for in` loop that iterates keys of objects or items of arrays and
+a counting `for` loop that is a variation of the `while` loop.
+
+```javascript
+{%
+
+  i = 0;
+  arr = [1, 2, 3];
+  obj = { Alice: 32, Bob: 54 };
+
+  // execute as long as condition is true
+  while (i < length(arr)) {
+      print(arr[i], "\n");
+      i++;
+  }
+
+  // execute for each item in arr
+  for (n in arr) {
+      print(n, "\n");
+  }
+
+  // execute for each key in obj
+  for (person in obj) {
+      print(person, " is ", obj[person], " years old.\n");
+  }
+
+  // execute initialization statement (j = 0) once
+  // execute as long as condition (j < length(arr)) is true
+  // execute step statement (j++) after each iteration
+  for (j = 0; j < length(arr); j++) {
+      print(arr[j], "\n");
+  }
+
+%}
+```
+
+##### 3.3. Alternative syntax
+
+Since conditional statements and loops are often used for template formatting
+purposes, e.g. to repeat a specific markup for each item of a list, ucode
+supports an alternative syntax that does not require curly braces to group
+statements but that uses explicit end keywords to denote the end of the control
+statement body for better readability instead.
+
+The following two examples first illustrate the normal syntax, followed by the
+alternative syntax that is more suitable for statement blocks:
+
+```
+Printing a list:
+{% for (n in [1, 2, 3]) { -%}
+  - Item #{{ n }}
+{% } %}
+```
+
+The alternative syntax replaces the opening curly brace (`{`) with a colon
+(`:`) and the closing curly brace (`}`) with an explicit `endfor` keyword:
+
+```
+Printing a list:
+{% for (n in [1, 2, 3]): -%}
+  - Item #{{ n }}
+{% endfor %}
+```
+
+For each control statement type, a corresponding alternative end keyword is defined:
+
+  - `if (...): ... endif`
+  - `for (...): ... endfor`
+  - `while (...): ... endwhile`
+
+
+#### 4. Functions
+
+Ucode scripts may define functions to group repeating operations into reusable
+operations. Functions can be both declared with a name, in which case they're
+automatically registered in the current scope, or anonymously which allows
+assigning the resulting value to a variable, e.g. to build arrays or objects of
+functions:
+
+```javascript
+{%
+
+  function duplicate(n) {
+       return n * 2;
+  }
+
+  let utilities = {
+      concat: function(a, b) {
+          return "" + a + b;
+      },
+      greeting: function() {
+          return "Hello, " + getenv("USER") + "!";
+      }
+  };
+
+-%}
+
+The duplicate of 2 is {{ duplicate(2) }}.
+The concatenation of 'abc' and 123 is {{ utilities.concat("abc", 123) }}.
+Your personal greeting is: {{ utilities.greeting() }}.
+```
+
+##### 4.1. Alternative syntax
+
+Function declarations support the same kind of alternative syntax as defined
+for control statements (3.3.)
+
+The alternative syntax replaces the opening curly brace (`{`) with a colon
+(`:`) and the closing curly brace (`}`) with an explicit `endfunction`
+keyword:
+
+```
+{% function printgreeting(name): -%}
+  Hallo {{ name }}, nice to meet you.
+{% endfunction -%}
+
+<h1>{{ printgreeting("Alice") }}</h1>
+```
+
+
+#### 5. Operators
+
+Similar to JavaScript and C, ucode scripts support a range of different
+operators to manipulate values and variables.
+
+##### 5.1. Arithmetic operations
+
+The operators `+`, `-`, `*`, `/`, `%`, `++` and `--` allow to perform
+additions, substractions, multiplications, divisions, modulo, increment or
+decrement operations respectively where the result depends on the type of
+involved values.
+
+The `++` and `--` operators are unary, means that they only apply to one
+operand. The `+` and `-` operators may be used in unary context to either
+convert a given value to a numeric value or to negate a given value.
+
+If either operand of the `+` operator is a string, the other one is converted
+to a string value as well and a concatenated string is returned.
+
+All other arithmetic operators coerce their operands into numeric values.
+Fractional values are converted to doubles, other numeric values to integers.
+
+If either operand is a double, the other one is converted to a double value as
+well and a double result is returned.
+
+Divisions by zero result in the special double value `Infinity`. If an operand
+cannot be converted to a numeric value, the result of the operation is the
+special double value `NaN`.
+
+```javascript
+{%
+  a = 2;
+  b = 5.2;
+  s1 = "125";
+  s2 = "Hello world";
+
+  print(+s1);      // 125
+  print(+s2);      // NaN
+  print(-s1);      // -125
+  print(-s2);      // NaN
+  print(-a);       // -2
+
+  print(a++);      // 2 (Return value of a, then increment by 1)
+  print(++a);      // 4 (Increment by 1, then return value of a)
+
+  print(b--);      // 5.2 (Return value of b, then decrement by 1)
+  print(--b);      // 3.2 (Decrement by 1, then return value of b)
+
+  print(4 + 8);    // 12
+  print(7 - 4);    // 3
+  print(3 * 3);    // 9
+
+  print(10 / 4);   // 2 (Integer division)
+  print(10 / 4.0); // 2.5 (Double division)
+  print(10 / 0);   // Infinity
+
+  print(10 % 7);   // 3
+  print(10 % 7.0); // NaN (Modulo is undefined for non-integers)
+%}
+```
+
+##### 5.2. Bitwise operations
+
+The operators `&`, `|`, `^`, `<<`, `>>` and `~` allow to perform bitwise and,
+or, xor, left shift, right shift and complement operations respectively.
+
+The `~` operator is unary, means that is only applies to one operand.
+
+```javascript
+{%
+  print(0 & 0, 0 & 1, 1 & 1);  // 001
+  print(0 | 0, 0 | 1, 1 | 1);  // 011
+  print(0 ^ 0, 0 ^ 1, 1 ^ 1);  // 010
+  print(10 << 2);              // 40
+  print(10 >> 2);              // 2
+  print(~15);                  // -16 (0xFFFFFFFFFFFFFFF0)
+%}
+```
+
+An important property of bitwise operators is that they're coercing their
+operand values to whole integers:
+
+```javascript
+{%
+  print(12.34 >> 0);   // 12
+  print(~(~12.34));    // 12
+%}
+```
+
+##### 5.3. Relational operations
+
+The operators `==`, `!=`, `<`, `<=`, `>` and `>=` test whether their operands
+are equal, inequal, lower than, lower than/equal to, higher than or higher
+than/equal to each other respectively.
+
+If both operands are strings, their respective byte values are compared, if
+both are objects or arrays, their underlying memory addresses are compared.
+
+In all other cases, both operands are coerced into numeric values and the
+resulting values are compared with each other.
+
+This means that comparing values of different types will coerce them both to
+numbers.
+
+The result of the relational operation is a boolean indicating truishness.
+
+```javascript
+{%
+  print(123 == 123);     // true
+  print(123 == "123");   // true!
+  print(123 < 456);      // true
+  print(123 > 456);      // false
+  print(123 != 456);     // true
+  print(123 != "123");   // false!
+  print({} == {});       // false (two different anonymous objects)
+  a = {}; print(a == a); // true (same object)
+%}
+```
+
+##### 5.4. Logical operations
+
+The operators `&&`, `||`, `??` and `!` test whether their operands are all true,
+partially true, null or false respectively.
+
+In the case of `&&` the rightmost value is returned while `||` results in the
+first truish and `??` in the first non-null value.
+
+The unary `!` operator will result in `true` if the operand is not trueish,
+otherwise it will result in `false`.
+
+Operands are evaluated from left to right while testing truishness, which means
+that expressions with side effects, such as function calls, are only executed
+if the preceeding condition was satisifed.
+
+```javascript
+{%
+  print(1 && 2 && 3);                 // 3
+  print(1 || 2 || 3);                 // 1
+  print(2 > 1 && 3 < 4);              // true
+  print(doesnotexist ?? null ?? 42);  // 42
+  print(1 ?? 2 ?? 3);                 // 1
+  print(!false);                      // true
+  print(!true);                       // false
+
+  res = test1() && test2();  // test2() is only called if test1() returns true
+%}
+```
+
+##### 5.5. Assignment operations
+
+In addition to the basic assignment operator `=`, most other operators have a
+corresponding shortcut assignment operator which reads the specified variable,
+applies the operation and operand to it, and writes it back.
+
+The result of assignment expressions is the assigned value.
+
+```javascript
+{%
+  a = 1;     // assign 1 to variable a
+  a += 2;    // a = a + 2;
+  a -= 3;    // a = a - 3;
+  a *= 4;    // a = a * 4;
+  a /= 5;    // a = a / 5;
+  a %= 6;    // a = a % 6;
+  a &= 7;    // a = a & 7;
+  a |= 8;    // a = a | 8;
+  a ^= 9;    // a = a ^ 9;
+  a <<= 10;  // a = a << 10;
+  a >>= 11;  // a = a >> 11;
+  a &&= 12;  // a = a && 12;
+  a ||= 13;  // a = a || 13;
+  a ??= 14;  // a = a ?? 14;
+
+  print(a = 2);  // 2
+%}
+```
+
+##### 5.6. Miscellaneous operators
+
+Besides the operators described so far, ucode script also supports a `delete`
+operator which removes a property from an object value.
+
+```javascript
+{%
+  a = { test: true };
+
+  delete a.test;         // true
+  delete a.notexisting;  // false
+
+  print(a);              // { }
+%}
+```
diff --git a/docs/ucode.css b/docs/ucode.css
new file mode 100644
index 0000000..a40f478
--- /dev/null
+++ b/docs/ucode.css
@@ -0,0 +1,20 @@
+#sidebar > .sidebar-items-container {
+	margin-top: 1rem;
+}
+
+h1 {
+	display: flex;
+}
+
+article .description > ul,
+article .description > ol {
+	list-style: initial;
+}
+
+.navbar-item.github-home a {
+	background-image: url("data:image/svg+xml,%3Csvg width='98' height='96' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath fill-rule='evenodd' clip-rule='evenodd' d='M48.854 0C21.839 0 0 22 0 49.217c0 21.756 13.993 40.172 33.405 46.69 2.427.49 3.316-1.059 3.316-2.362 0-1.141-.08-5.052-.08-9.127-13.59 2.934-16.42-5.867-16.42-5.867-2.184-5.704-5.42-7.17-5.42-7.17-4.448-3.015.324-3.015.324-3.015 4.934.326 7.523 5.052 7.523 5.052 4.367 7.496 11.404 5.378 14.235 4.074.404-3.178 1.699-5.378 3.074-6.6-10.839-1.141-22.243-5.378-22.243-24.283 0-5.378 1.94-9.778 5.014-13.2-.485-1.222-2.184-6.275.486-13.038 0 0 4.125-1.304 13.426 5.052a46.97 46.97 0 0 1 12.214-1.63c4.125 0 8.33.571 12.213 1.63 9.302-6.356 13.427-5.052 13.427-5.052 2.67 6.763.97 11.816.485 13.038 3.155 3.422 5.015 7.822 5.015 13.2 0 18.905-11.404 23.06-22.324 24.283 1.78 1.548 3.316 4.481 3.316 9.126 0 6.6-.08 11.897-.08 13.526 0 1.304.89 2.853 3.316 2.364 19.412-6.52 33.405-24.935 33.405-46.691C97.707 22 75.788 0 48.854 0z' fill='%2324292f'/%3E%3C/svg%3E");
+	background-repeat: no-repeat;
+	background-size: 1.5em;
+	background-position: 1rem center;
+	padding-left: 3rem;
+}
diff --git a/docs/ucode.js b/docs/ucode.js
new file mode 100644
index 0000000..35d2d23
--- /dev/null
+++ b/docs/ucode.js
@@ -0,0 +1,8 @@
+document.addEventListener('DOMContentLoaded', (ev) => {
+	const accordionState = window.localStorage.getItem('accordion-id');
+
+	if (accordionState == null || accordionState == '{}')
+		document
+			.querySelector('[data-isopen="false"]')
+			.setAttribute('data-isopen', 'true');
+});