path: root/docs/README.md

# The ucode Scripting Language

The ucode language is a tiny general purpose scripting language featuring a
syntax closely resembling ECMAScript. It can be used in a stand-alone manner
by using the ucode command line interpreter or embedded into host applications
by linking libucode and utilizing its C language API. Additionally, ucode can
be invoked in template mode where control flow and expression logic statements
are embedded in Jinja-like markup blocks.

Besides aiming for small size, the major design goals of ucode are the ability
to trivially read and write JSON data, good embeddability into C applications,
template capabilities for output formatting, extensiblity through loadable
native extension modules and a straightforward set of built-in functions
mimicking those found in the Perl 5 language.

## History and Motivation

In spring 2021 it has been decided to rewrite the OpenWrt firewall framework on
top of nftables with the goal to replace the then current C application with a
kind of preprocessor generating nftables rulesets using a set of templates
instead of relying on built-in hardcoded rules like its predecessor.

That decision spurred the development of *ucode*, initially meant to be a
simple template processor solely for the OpenWrt nftables firewall but quickly
evolving into a general purpose scripting language suitable for a wider range
of system scripting tasks.

Despite OpenWrt predominantly relying on POSIX shell and Lua as system
scripting languages already, a new solution was needed to accomodate the needs
of the new firewall implementation; mainly the ability to efficiently deal with
JSON data and complex data structures such as arrays and dictionaries and the
ability to closely interface with OpenWrt's *ubus* message bus system.

Throughout the design process of the new firewall and its template processor,
the following design goals were defined for the *ucode* scripting language:

 - Ability to embed code logic fragments such as control flow statements,
   function calls or arithmetic expressions into plain text templates, using
   a block syntax and functionality roughly inspired by Jinja templates
 - Built-in support for JSON data parsing and serialization, without the need
   for external libraries
 - Distinct array and object types (compared to Lua's single table datatype)
 - Distinct integer and float types and guaranteed 64bit integer range
 - Built-in support for bit operations
 - Built-in support for (POSIX) regular expressions
 - A comprehensive set of built-in standard functions, inspired by the core
   functions found in the Perl 5 interpreter
 - Staying as close to ECMAScript syntax as possible due to higher developer
   familiarity and to be able to reuse existing tooling such as editor syntax
   highlighting
 - Bindings for all relevant Linux and OpenWrt APIs, such as *ubus*, *uci*,
   *uloop*, *netlink* etc.
 - Procedural, synchronous programming flow
 - Very small executable size (the interpreter and runtime is currently around
   64KB on ARM Cortex A9)
 - Embeddability into C host applications

Summarized, *ucode* can be described as synchronous ECMAScript without the
object oriented standard library.


## Installation

### OpenWrt

In OpenWrt 22.03 and later, *ucode* should already be preinstalled. If not,
it can be installed via the package manager, using the `opkg install ucode`
command.

### MacOS

To build on MacOS, first install *cmake* and *json-c* via
[Homebrew](https://brew.sh/), then clone the ucode repository and execute
*cmake* followed by *make*:

    $ brew install cmake json-c
    $ git clone https://github.com/jow-/ucode.git
    $ cd ucode/
    $ cmake -DUBUS_SUPPORT=OFF -DUCI_SUPPORT=OFF -DULOOP_SUPPORT=OFF .
    $ make
    $ sudo make install

### Debian

The ucode repository contains build recipes for Debian packages, to build .deb
packages for local installation, first install required development packages,
then clone the repository and invoke *dpkg-buildpackage* to produce the binary
package files:

    $ sudo apt-get install build-essential devscripts debhelper libjson-c-dev cmake pkg-config
    $ git clone https://github.com/jow-/ucode.git
    $ cd ucode/
    $ dpkg-buildpackage -b -us -uc
    $ sudo dpkg -i ../ucode*.deb ../libucode*.deb

### Other Linux systems

To install ucode from source on other systems, ensure that the json-c library
and associated development headers are installed, then clone and compile the
ucode repository:

    $ git clone https://github.com/jow-/ucode.git
    $ cd ucode/
    $ cmake -DUBUS_SUPPORT=OFF -DUCI_SUPPORT=OFF -DULOOP_SUPPORT=OFF .
    $ make
    $ sudo make install


## Syntax

### Template mode

By default, *ucode* is executed in *raw mode*, means it expects a given source
file to only contain script code. By invoking the ucode interpreter with the
`-T` flag or by using the `utpl` alias, the *ucode* interpreter is switched
into *template mode* where the source file is expected to be a plaintext file
containing *template blocks* containing ucode script expressions or comments.

#### Block types

There are three kinds of blocks; *expression blocks*, *statement blocks* and
*comment blocks*. The former two embed code logic using ucode's JavaScript-like
syntax while the latter comment block type is simply discarded during
processing.


##### 1. Statement block

Statement blocks are enclosed in an opening `{%` and a closing `%}` tag and
may contain any number of script code statements, even entire programs.

It is allowed to omit the closing `%}` of a statement block to parse the
entire remaining source text after the opening tag as ucode script.

By default, statement blocks produce no output and the entire block is
reduced to an empty string during template evaluation but contained script
code might invoke functions such as `print()` to explicitly output contents.

For example the following template would result in `The epoch is odd` or
`The epoch is even`, depending on the current epoch value:

`The epoch is {% if (time() % 2): %}odd{% else %}even{% endif %}!`


##### 2. Expression block

Expression blocks are enclosed in an opening `{{` and a closing `}}` tag and
may only contain a single expression statement (multiple expressions may be
chained with comma). The implicit result of the rightmost evaluated expression
is used as output when processing the block.

For example the template `Hello world, {{ getenv("USER") }}!` would result in
the output "Hello world, user!" where `user` would correspond to the name of
the current user executing the ucode interpreter.


##### 3. Comment block

Comment blocks, which are denoted with an opening `{#` and a closing `#}` tag
may contain arbitrary text except the closing `#}` tag itself. Comments blocks
are completely stripped during processing and are replaced with an empty string.

The following example template would result in the output "Hello world":

`Hello {# mad #}word`


#### Whitespace handling

Each block start tag may be suffixed with a dash to strip any whitespace
before the block and likewise any block end tag may be prefixed with a dash
to strip any whitespace following the block.

Without using whitespace stripping, the following example:

```
This is a first line
{% for (x in [1, 2, 3]): %}
This is item {{ x }}.
{% endfor %}
This is the last line
```

Would result in the following output:

```
This is a first line

This is item 1.
This is item 2.
This is item 3.

This is the last line
```

By adding a trailing dash to apply whitespace stripping after the block, the
empty lines can be eliminated:

```
This is a first line
{% for (x in [1, 2, 3]): -%}
This is item {{ x }}.
{% endfor -%}
This is the last line
```

Output:

```
This is a first line
This is item 1.
This is item 2.
This is item 3.
This is the last line
```

By applying whitespace stripping before the block, all lines can be joined
into a single output line:

```
This is a first line
{%- for (x in [1, 2, 3]): -%}
This is item {{ x }}.
{%- endfor -%}
This is the last line
```

Output:

```
This is a first lineThis is item 1.This is item 2.This is item 3.This is the last line
```

### Script syntax

The ucode script language - used either within statement and expression blocks
or throughout the entire file in *raw mode*, uses untyped variables and employs
a simplified JavaScript like syntax.

The language implements function scoping and differentiates between local and
global variables. Each function has its own private scope while executing and
local variables declared inside a function are not accessible in the outer
calling scope.

#### 1. Data types

Ucode supports seven different basic types as well as two additional special
types; function values and ressource values. The supported types are:

 - Boolean values (`true` or `false`)
 - Integer values (`-9223372036854775808` to `+9223372036854775807`)
 - Double values (`-1.7e308` to `+1.7e308`)
 - String values (e.g. `'Hello world!'` or `"Sunshine \u2600!"`)
 - Array values (e.g. `[1, false, "foo"]`)
 - Object values (e.g. `{ foo: true, "bar": 123 }`)
 - Null value (`null`)

Ucode utilizes reference counting to manage memory used for variables and values
and frees data automatically as soon as values go out of scope.

Numeric values are either stored as signed 64bit integers or as IEEE 756 double
value. Conversion between integer and double values can happen implicitly, e.g.
through numeric operations, or explicitely, e.g. by invoking functions such as
`int()`.

#### 2. Variables

Variable names must start with a letter or an underscore and may only contain
the characters `A`..`Z`, `a`..`z`, `0`..`9` or `_`. By prefixing a variable
name with the keyword `let`, it is declared in the local block scope only
and not visible outside anymore.

Variables may also be declared using the `const` keyword. Such variables follow
the same scoping rules as `let` declared ones but they cannot be modified after
they have been declared. Any attempt to do so will result in a syntax error
during compilation.

```javascript
{%

  a = 1;  // global variable assignment

  function test() {
    let b = 2;  // declare `b` as local variable
    a = 2;        // overwrite global a
  }

  test();

  print(a, "\n");  // outputs "2"
  print(b, "\n");  // outputs nothing

  const c = 3;
  print(c, "\n");  // outputs "3"

  c = 4;           // raises syntax error
  c++;             // raises syntax error

  const d;         // raises syntax error, const variables must
                   // be initialized at declaration time

%}
```

#### 3. Control statements

Similar to JavaScript, ucode supports `if`, `for` and `while` statements to
control execution flow.

##### 3.1. Conditional statement

If/else blocks can be used to execute statements depending on a condition.

```javascript
{%

  user = getenv("USER");

  if (user == "alice") {
      print("Hello Alice!\n");
  }
  else if (user == "bob") {
      print("Hello Bob!\n");
  }
  else {
      print("Hello guest!\n");
  }

%}
```

If only a single statement is wrapped by an if or else branch, the enclosing
curly braces may be omitted:

```javascript
{%

  if (rand() == 3)
      print("This is quite unlikely\n");

%}
```

##### 3.2. Loop statements

Ucode script supports three different flavors of loop control statements; a
`while` loop that executes enclosed statements as long as the loop condition is
fulfilled, a `for in` loop that iterates keys of objects or items of arrays and
a counting `for` loop that is a variation of the `while` loop.

```javascript
{%

  i = 0;
  arr = [1, 2, 3];
  obj = { Alice: 32, Bob: 54 };

  // execute as long as condition is true
  while (i < length(arr)) {
      print(arr[i], "\n");
      i++;
  }

  // execute for each item in arr
  for (n in arr) {
      print(n, "\n");
  }

  // execute for each key in obj
  for (person in obj) {
      print(person, " is ", obj[person], " years old.\n");
  }

  // execute initialization statement (j = 0) once
  // execute as long as condition (j < length(arr)) is true
  // execute step statement (j++) after each iteration
  for (j = 0; j < length(arr); j++) {
      print(arr[j], "\n");
  }

%}
```

##### 3.3. Alternative syntax

Since conditional statements and loops are often used for template formatting
purposes, e.g. to repeat a specific markup for each item of a list, ucode
supports an alternative syntax that does not require curly braces to group
statements but that uses explicit end keywords to denote the end of the control
statement body for better readability instead.

The following two examples first illustrate the normal syntax, followed by the
alternative syntax that is more suitable for statement blocks:

```
Printing a list:
{% for (n in [1, 2, 3]) { -%}
  - Item #{{ n }}
{% } %}
```

The alternative syntax replaces the opening curly brace (`{`) with a colon
(`:`) and the closing curly brace (`}`) with an explicit `endfor` keyword:

```
Printing a list:
{% for (n in [1, 2, 3]): -%}
  - Item #{{ n }}
{% endfor %}
```

For each control statement type, a corresponding alternative end keyword is defined:

  - `if (...): ... endif`
  - `for (...): ... endfor`
  - `while (...): ... endwhile`


#### 4. Functions

Ucode scripts may define functions to group repeating operations into reusable
operations. Functions can be both declared with a name, in which case they're
automatically registered in the current scope, or anonymously which allows
assigning the resulting value to a variable, e.g. to build arrays or objects of
functions:

```javascript
{%

  function duplicate(n) {
       return n * 2;
  }

  let utilities = {
      concat: function(a, b) {
          return "" + a + b;
      },
      greeting: function() {
          return "Hello, " + getenv("USER") + "!";
      }
  };

-%}

The duplicate of 2 is {{ duplicate(2) }}.
The concatenation of 'abc' and 123 is {{ utilities.concat("abc", 123) }}.
Your personal greeting is: {{ utilities.greeting() }}.
```

##### 4.1. Alternative syntax

Function declarations support the same kind of alternative syntax as defined
for control statements (3.3.)

The alternative syntax replaces the opening curly brace (`{`) with a colon
(`:`) and the closing curly brace (`}`) with an explicit `endfunction`
keyword:

```
{% function printgreeting(name): -%}
  Hallo {{ name }}, nice to meet you.
{% endfunction -%}

<h1>{{ printgreeting("Alice") }}</h1>
```


#### 5. Operators

Similar to JavaScript and C, ucode scripts support a range of different
operators to manipulate values and variables.

##### 5.1. Arithmetic operations

The operators `+`, `-`, `*`, `/`, `%`, `++` and `--` allow to perform
additions, substractions, multiplications, divisions, modulo, increment or
decrement operations respectively where the result depends on the type of
involved values.

The `++` and `--` operators are unary, means that they only apply to one
operand. The `+` and `-` operators may be used in unary context to either
convert a given value to a numeric value or to negate a given value.

If either operand of the `+` operator is a string, the other one is converted
to a string value as well and a concatenated string is returned.

All other arithmetic operators coerce their operands into numeric values.
Fractional values are converted to doubles, other numeric values to integers.

If either operand is a double, the other one is converted to a double value as
well and a double result is returned.

Divisions by zero result in the special double value `Infinity`. If an operand
cannot be converted to a numeric value, the result of the operation is the
special double value `NaN`.

```javascript
{%
  a = 2;
  b = 5.2;
  s1 = "125";
  s2 = "Hello world";

  print(+s1);      // 125
  print(+s2);      // NaN
  print(-s1);      // -125
  print(-s2);      // NaN
  print(-a);       // -2

  print(a++);      // 2 (Return value of a, then increment by 1)
  print(++a);      // 4 (Increment by 1, then return value of a)

  print(b--);      // 5.2 (Return value of b, then decrement by 1)
  print(--b);      // 3.2 (Decrement by 1, then return value of b)

  print(4 + 8);    // 12
  print(7 - 4);    // 3
  print(3 * 3);    // 9

  print(10 / 4);   // 2 (Integer division)
  print(10 / 4.0); // 2.5 (Double division)
  print(10 / 0);   // Infinity

  print(10 % 7);   // 3
  print(10 % 7.0); // NaN (Modulo is undefined for non-integers)
%}
```

##### 5.2. Bitwise operations

The operators `&`, `|`, `^`, `<<`, `>>` and `~` allow to perform bitwise and,
or, xor, left shift, right shift and complement operations respectively.

The `~` operator is unary, means that is only applies to one operand.

```javascript
{%
  print(0 & 0, 0 & 1, 1 & 1);  // 001
  print(0 | 0, 0 | 1, 1 | 1);  // 011
  print(0 ^ 0, 0 ^ 1, 1 ^ 1);  // 010
  print(10 << 2);              // 40
  print(10 >> 2);              // 2
  print(~15);                  // -16 (0xFFFFFFFFFFFFFFF0)
%}
```

An important property of bitwise operators is that they're coercing their
operand values to whole integers:

```javascript
{%
  print(12.34 >> 0);   // 12
  print(~(~12.34));    // 12
%}
```

##### 5.3. Relational operations

The operators `==`, `!=`, `<`, `<=`, `>` and `>=` test whether their operands
are equal, inequal, lower than, lower than/equal to, higher than or higher
than/equal to each other respectively.

If both operands are strings, their respective byte values are compared, if
both are objects or arrays, their underlying memory addresses are compared.

In all other cases, both operands are coerced into numeric values and the
resulting values are compared with each other.

This means that comparing values of different types will coerce them both to
numbers.

The result of the relational operation is a boolean indicating truishness.

```javascript
{%
  print(123 == 123);     // true
  print(123 == "123");   // true!
  print(123 < 456);      // true
  print(123 > 456);      // false
  print(123 != 456);     // true
  print(123 != "123");   // false!
  print({} == {});       // false (two different anonymous objects)
  a = {}; print(a == a); // true (same object)
%}
```

##### 5.4. Logical operations

The operators `&&`, `||`, `??` and `!` test whether their operands are all true,
partially true, null or false respectively.

In the case of `&&` the rightmost value is returned while `||` results in the
first truish and `??` in the first non-null value.

The unary `!` operator will result in `true` if the operand is not trueish,
otherwise it will result in `false`.

Operands are evaluated from left to right while testing truishness, which means
that expressions with side effects, such as function calls, are only executed
if the preceeding condition was satisifed.

```javascript
{%
  print(1 && 2 && 3);                 // 3
  print(1 || 2 || 3);                 // 1
  print(2 > 1 && 3 < 4);              // true
  print(doesnotexist ?? null ?? 42);  // 42
  print(1 ?? 2 ?? 3);                 // 1
  print(!false);                      // true
  print(!true);                       // false

  res = test1() && test2();  // test2() is only called if test1() returns true
%}
```

##### 5.5. Assignment operations

In addition to the basic assignment operator `=`, most other operators have a
corresponding shortcut assignment operator which reads the specified variable,
applies the operation and operand to it, and writes it back.

The result of assignment expressions is the assigned value.

```javascript
{%
  a = 1;     // assign 1 to variable a
  a += 2;    // a = a + 2;
  a -= 3;    // a = a - 3;
  a *= 4;    // a = a * 4;
  a /= 5;    // a = a / 5;
  a %= 6;    // a = a % 6;
  a &= 7;    // a = a & 7;
  a |= 8;    // a = a | 8;
  a ^= 9;    // a = a ^ 9;
  a <<= 10;  // a = a << 10;
  a >>= 11;  // a = a >> 11;
  a &&= 12;  // a = a && 12;
  a ||= 13;  // a = a || 13;
  a ??= 14;  // a = a ?? 14;

  print(a = 2);  // 2
%}
```

##### 5.6. Miscellaneous operators

Besides the operators described so far, ucode script also supports a `delete`
operator which removes a property from an object value.

```javascript
{%
  a = { test: true };

  delete a.test;         // true
  delete a.notexisting;  // false

  print(a);              // { }
%}
```