go_marshal: Implement automatic generation of ABI marshalling code.

This CL implements go_marshal, a code generation utility for
automatically serializing and deserializing ABI structs.

The go_marshal tool automatically generates implementations of the new
marshal interface. Unlike binary.Marshal/Unmarshal, the generated
interface implementations use no runtime reflection, and translates to
a single memcpy for most structs. See go_marshal/README.md for
details.

PiperOrigin-RevId: 268065475

1 files changed, 164 insertions, 0 deletions

diff --git a/tools/go_marshal/README.md b/tools/go_marshal/README.md
new file mode 100644
index 000000000..481575bd3
--- /dev/null
+++ b/tools/go_marshal/README.md
@@ -0,0 +1,164 @@
+This package implements the go_marshal utility.
+
+# Overview
+
+`go_marshal` is a code generation utility similar to `go_stateify` for
+automatically generating code to marshal go data structures to memory.
+
+`go_marshal` attempts to improve on `binary.Write` and the sentry's
+`binary.Marshal` by moving the go runtime reflection necessary to marshal a
+struct to compile-time.
+
+`go_marshal` automatically generates implementations for `abi.Marshallable` and
+`safemem.{Reader,Writer}`. Call-sites for serialization (typically syscall
+implementations) can directly invoke `safemem.Reader.ReadToBlocks` and
+`safemem.Writer.WriteFromBlocks`. Data structures that require custom
+serialization will have manual implementations for these interfaces.
+
+Data structures can be flagged for code generation by adding a struct-level
+comment `// +marshal`.
+
+# Usage
+
+See `defs.bzl`: two new rules are provided, `go_marshal` and `go_library`.
+
+The recommended way to generate a go library with marshalling is to use the
+`go_library` with mostly identical configuration as the native go_library rule.
+
+```
+load("<PKGPATH>/gvisor/tools/go_marshal:defs.bzl", "go_library")
+
+go_library(
+    name = "foo",
+    srcs = ["foo.go"],
+)
+```
+
+Under the hood, the `go_marshal` rule is used to generate a file that will
+appear in a Go target; the output file should appear explicitly in a srcs list.
+For example (note that the above is the preferred method):
+
+```
+load("<PKGPATH>/gvisor/tools/go_marshal:defs.bzl", "go_marshal")
+
+go_marshal(
+    name = "foo_abi",
+    srcs = ["foo.go"],
+    out = "foo_abi.go",
+    package = "foo",
+)
+
+go_library(
+    name = "foo",
+    srcs = [
+        "foo.go",
+        "foo_abi.go",
+    ],
+    deps = [
+        "<PKGPATH>/gvisor/pkg/abi",
+        "<PKGPATH>/gvisor/pkg/sentry/safemem/safemem",
+        "<PKGPATH>/gvisor/pkg/sentry/usermem/usermem",
+    ],
+)
+```
+
+As part of the interface generation, `go_marshal` also generates some tests for
+sanity checking the struct definitions for potential alignment issues, and a
+simple round-trip test through Marshal/Unmarshal to verify the implementation.
+These tests use reflection to verify properties of the ABI struct, and should be
+considered part of the generated interfaces (but are too expensive to execute at
+runtime). Ensure these tests run at some point.
+
+```
+$ cat BUILD
+load("<PKGPATH>/gvisor/tools/go_marshal:defs.bzl", "go_library")
+
+go_library(
+    name = "foo",
+    srcs = ["foo.go"],
+)
+$ blaze build :foo
+$ blaze query ...
+<path-to-dir>:foo_abi_autogen
+<path-to-dir>:foo_abi_autogen_test
+$ blaze test :foo_abi_autogen_test
+<test-output>
+```
+
+# Restrictions
+
+Not all valid go type definitions can be used with `go_marshal`. `go_marshal` is
+intended for ABI structs, which have these additional restrictions:
+
+-   At the moment, `go_marshal` only supports struct declarations.
+
+-   Structs are marshalled as packed types. This means no implicit padding is
+    inserted between fields shorter than the platform register size. For
+    alignment, manually insert padding fields.
+
+-   Structs used with `go_marshal` must have a compile-time static size. This
+    means no dynamically sizes fields like slices or strings. Use statically
+    sized array (byte arrays for strings) instead.
+
+-   No pointers, channel, map or function pointer fields, and no fields that are
+    arrays of these types. These don't make sense in an ABI data structure.
+
+-   We could support opaque pointers as `uintptr`, but this is currently not
+    implemented. Implementing this would require handling the architecture
+    dependent native pointer size.
+
+-   Fields must either be a primitive integer type (`byte`,
+    `[u]int{8,16,32,64}`), or of a type that implements abi.Marshallable.
+
+-   `int` and `uint` fields are not allowed. Use an explicitly-sized numeric
+    type.
+
+-   `float*` fields are currently not supported, but could be if necessary.
+
+# Appendix
+
+## Working with Non-Packed Structs
+
+ABI structs must generally be packed types, meaning they should have no implicit
+padding between short fields. However, if a field is tagged
+`marshal:"unaligned"`, `go_marshal` will fall back to a safer but slower
+mechanism to deal with potentially unaligned fields.
+
+Note that the non-packed property is inheritted by any other struct that embeds
+this struct, since the `go_marshal` tool currently can't reason about alignments
+for embedded structs that are not aligned.
+
+Because of this, it's generally best to avoid using `marshal:"unaligned"` and
+insert explicit padding fields instead.
+
+## Debugging go_marshal
+
+To enable debugging output from the go marshal tool, pass the `-debug` flag to
+the tool. When using the build rules from above, add a `debug = True` field to
+the build rule like this:
+
+```
+load("<PKGPATH>/gvisor/tools/go_marshal:defs.bzl", "go_library")
+
+go_library(
+    name = "foo",
+    srcs = ["foo.go"],
+    debug = True,
+)
+```
+
+## Modifying the `go_marshal` Tool
+
+The following are some guidelines for modifying the `go_marshal` tool:
+
+-   The `go_marshal` tool currently does a single pass over all types requesting
+    code generation, in arbitrary order. This means the generated code can't
+    directly obtain information about embedded marshallable types at
+    compile-time. One way to work around this restriction is to add a new
+    Marshallable interface method providing this piece of information, and
+    calling it from the generated code. Use this sparingly, as we want to rely
+    on compile-time information as much as possible for performance.
+
+-   No runtime reflection in the code generated for the marshallable interface.
+    The entire point of the tool is to avoid runtime reflection. The generated
+    tests may use reflection.