| Age | Commit message (Collapse) | Author |
|---|
|
This is a suite of changes intended to dramatically speed up nogo speed.
First, there are minor changes that help efficiency significantly.
* Gob-based encoding is used internally, and JSON only used for the final
set of findings. This is done to preserve the existing format (which is
consumed by external tooling), and to facilitate manual debugging.
* Unnecessary regex compilation is elided in the configuration, and care is
taken for merges to prevent redundant entries. I'm not sure quite sure how,
but it turns out that this was consumed a significant amount of time,
presumably compiling the same regexes over and over again.
Second, this change enables bazel workers for nogo analyzers.
Workers enable persistent processes instead of creating and tearing down a
sandbox every invocation. A library is introduced to abstraction these details,
and allow the tools to still be written using standard flags, etc.
The key here is that these binaries and the core of nogo become aware of
caches with worker.Cache. This allows us to save significant time loading the
same set of files and findings over and over again. These caches are keyed by
the digests that are provided by bazel, and are capped in overall size.
Note that the worker package attempts to capture output during each run, but
tools are no longer permitted to write to stdout. This necessitated dropping
some spurious output from checklocks.
PiperOrigin-RevId: 370505732
|
|
Presently, the standard library facts are not serialized in a deterministic
order. This means that they have the possibility to change on each iteration,
requiring a large scale re-analysis of all downstream actions, which includes
all packages.
Improve cache-ability of nogo actions by improving the determinism of the both
facts and findings. Internally, default facts should be serialized as a sorted
list for this reason already.
PiperOrigin-RevId: 370188259
|
|
We already have blocking nogo tests which show all findings. This job was
building all nogo targets, and posting all the findings to GitHub as a check
run. Building nogo takes a while so we actually end up wasting a lot of time
doing redundant work.
This is aligned with our goal of moving away from GitHub actions to BuildKite
only.
PiperOrigin-RevId: 370134875
|
|
PiperOrigin-RevId: 369505182
|
|
gohacks.Memmove() takes in the number of bytes to move. The current generated
code passes len(src) and len(dst) as the number of bytes to move.
However, the marshal.Marshallable interface allows passing in larger buffers.
The stated precondition is that the buffer should be "at least" SizeBytes()
in length but it is allowed to be larger.
This change now correctly calls Memmove with the argument for the number of
bytes to move as type.SizeBytes(). This was caught when I made lisafs use the
Unsafe marshalling API and got a lot of memory violations.
PiperOrigin-RevId: 368952642
|
|
PiperOrigin-RevId: 368470656
|
|
Netstack is supposed to be somewhat independent of the rest of gVisor, and
others should be able to use it without pulling in excessive dependencies.
Currently, there is no way to fight dependency creep besides careful code
review.
This change introduces a test rule `netstack_deps_check` that ensures the target
only relies on gVisor targets and a short allowlist of external dependencies.
Users who add a dependency will see an error and have to manually update the
allowlist.
The set of packages to test comes from //runsc, as it uses packages we would
expect users to commonly rely on. It was generated via:
$ find ./runsc -name BUILD | xargs grep tcpip | awk '{print $2}' | sort | uniq
(Note: We considered giving //pkg/tcpip it's own go.mod, but this breaks go
tooling.)
PiperOrigin-RevId: 368456711
|
|
PiperOrigin-RevId: 367517305
|
|
PiperOrigin-RevId: 366555466
|
|
Split usermem package to help remove syserror dependency in go_marshal.
New hostarch package contains code not dependent on syserror.
PiperOrigin-RevId: 365651233
|
|
Stateify methods are always called without holding the appropriate
locks. The system is paused and we know there will be no mutations
when we call Save/Load, so this is perfectly safe. However, checklocks
can't know about this, and it will always complain.
Mark stateify generated methods that touch struct fields as
"checklocksignore" to avoid this.
PiperOrigin-RevId: 364610241
|
|
We were only supporting dynamic struct types. With this change, users can make
any type dynamic. The tool (correctly) blindly just generates the remaining
methods needed to implement Marshallable using the 3 methods defined by the
user on the dynamic type.
This is helpful in situations like:
type StringArray []string
Added a test for such a use case.
PiperOrigin-RevId: 364463164
|
|
This may be useful for tracking down where build tags come from and
understanding tag import issues in generated files.
PiperOrigin-RevId: 364374931
|
|
PiperOrigin-RevId: 362545342
|
|
We can generate more than one apt repo for the same package. If we will
sign a package again, its file will be changed and all hashes that have
been generated before will be invalid.
|
|
This binary is used to recursively enable and generate Merkle tree files
for all files and directories in a file system from inside a gVisor
sandbox.
PiperOrigin-RevId: 362418770
|
|
The dynamic type user defines the marshalling logic, so we don't need to test
for things like alignment, absence of slices, etc.
For dynamic types, the go_marshal generator just generates the missing methods
required to implement marshal.Marshallable.
PiperOrigin-RevId: 361676311
|
|
This validates that struct fields if annotated with "// checklocks:mu" where
"mu" is a mutex field in the same struct then access to the field is only
done with "mu" locked.
All types that are guarded by a mutex must be annotated with
// +checklocks:<mutex field name>
For more details please refer to README.md.
PiperOrigin-RevId: 360729328
|
|
This will fix debian packaging.
Updates #5510
PiperOrigin-RevId: 359563378
|
|
This makes it easier to implement dynamically sized types in go-marshal. You
really only need to implement MarshalBytes, UnmarshalBytes and SizeBytes to
implement the entire interface.
By using the `dynamic` tag, the autogenerator will generate the rest of the
methods for us.
This change also simplifies how KernelIPTGetEntries implements Marshallable
using the newly added utility.
PiperOrigin-RevId: 356397114
|
|
Confirmed gazelle generates no significant noise.
Fix documented gazelle invocation while I'm here.
PiperOrigin-RevId: 355452758
|
|
...and a bunch of other things as I worked through the rot. Notably:
- Upgrade to bazel 4.0.0
- Upgrade to Go 1.15.7
Remove go_branch stderr suppression; this made it quite difficult to see
what was failing while developing this patch.
PiperOrigin-RevId: 355257833
|
|
This is required only for the built-in bazel nogo functionality.
Since we roll these targets manually via the wrappers, we don't need
to use go_tool_library. The inconsistent use of these targets leads
to conflicting instantiations of go_default_library and go_tool_library,
which both contain the same output files.
PiperOrigin-RevId: 355184975
|
|
This change also adds an extra sanity check to the make_apt.sh script,
in order to ensure that this simple mistake does not occur again.
PiperOrigin-RevId: 355101754
|
|
It's unclear why permissions wind up corrupted, but these can be cleared
on any failure, similar to the bazel cache itself:
https://buildkite.com/gvisor/pipeline/builds/2304#_
PiperOrigin-RevId: 355057421
|
|
The GH Build action has been failing with the error message:
```
--- BUILD -c opt //runsc
tee: /proc/self/fd/2: No such device or address
```
tee /dev/fd/2 seems to be the canonical way of copying stdin to stderr.
So use that instead.
PiperOrigin-RevId: 353259087
|
|
Files removed from the working tree were not being properly removed from
the branch, leading to symbol conflicts while building. This requires the
change to 'git add --all' in the tools/go_branch.sh script.
But why was this not caught by CI? The "git clean -f" command by default
only cleans files in the current working directory. In order to clean the
whole tree recursively, we need to specify a pathspec, which is ".".
In addition to these fixes, re-add the "go tests" command to help prevent
this from happening again, since merges on the Go branch will happen in
GitHub actions for simplicity. The Go test is retained in BuildKite.
PiperOrigin-RevId: 351503804
|
|
gvisor-containerd-shim is not compatible with containerd 1.1 or earlier.
Starting from containerd 1.2, shim v2 is the preferred interface.
PiperOrigin-RevId: 351485556
|
|
This can race and cause issues. Instead, a manual shutdown can be done
via 'bazel-shutdown' if required for specific cases. The ARM64 builds
are now done using cross-compilation, so this hack is not necessary.
PiperOrigin-RevId: 351477908
|
|
These are primarily simplification and lint mistakes. However, minor
fixes are also included and tests added where appropriate.
PiperOrigin-RevId: 351425971
|
|
This requires several changes:
* Templates must preserve relevant tags.
* Pagetables templates are split into two targets, each preserving tags.
* The binary VDSO is similarly split into two targets, with some juggling.
* The top level tools/go_branch.sh now does a crossbuild of ARM64 as well,
and checks and merges the results of the two branches together.
Fixes #5178
PiperOrigin-RevId: 351304330
|
|
This change works around an issue in rules_pkg, described here:
https://github.com/bazelbuild/rules_pkg/pull/263
PiperOrigin-RevId: 350869030
|
|
make BAZEL_CONFIG=aarch64 arm-qemu-smoke-test
Signed-off-by: Andrei Vagin <avagin@gmail.com>
|
|
... Otherwise it will be mounted in via some other path, and will
not be available inside the container at all.
PiperOrigin-RevId: 350440843
|
|
We log a warning if objdump fails, but this appears in the build log, not test
log, which can make it hard to notice.
Include it with the actual escape output as context on "(possible)" to make it
more clear when something is wrong.
PiperOrigin-RevId: 350355759
|
|
This returns all targets, and handles no targets.
PiperOrigin-RevId: 350263578
|
|
PiperOrigin-RevId: 350200437
|
|
For validation, the "on" key in existing YAML files is changed to a literal
string. In the YAML spec, on is a keyword which encodes a boolean value, so
without relying on a specific implementation the YAML files are technically
not encoding an object that complies with the specification.
PiperOrigin-RevId: 350172147
|
|
PiperOrigin-RevId: 349616845
|
|
This change cleans up some minor Makefile issues, and adds support for
BuildKite annotations on failure and on profiles being generated. These
annotations will make failures very clear and link to the artifacts.
This change is a stepping stone for aggregating coverage data from all
individual test jobs, as this will also happen in .buildkite/annotate.sh.
PiperOrigin-RevId: 349606598
|
|
Otherwise this pollutes the 'go' branch and doesn't conform to standards
for generate bazel files.
PiperOrigin-RevId: 349605037
|
|
PiperOrigin-RevId: 349491589
|
|
Signed-off-by: Lai Jiangshan <jiangshan.ljs@antfin.com>
|
|
PiperOrigin-RevId: 347047550
|
|
PiperOrigin-RevId: 347038652
|
|
PiperOrigin-RevId: 346878344
|
|
A few images were broken with respect to aarch64. We should now
be able to run push-all-images with ARCH=aarch64 as part of the
regular continuous integration builds, and add aarch64 smoke tests
(via user emulation for now) to the regular test suite (future).
PiperOrigin-RevId: 346685462
|
|
AtomicPtrMap is a generic concurrent map from arbitrary keys to arbitrary
pointer values.
Benchmarks:
name time/op
StoreDelete/RWMutexMap-12 335ns ± 1%
StoreDelete/SyncMap-12 705ns ± 3%
StoreDelete/AtomicPtrMap-12 287ns ± 4%
StoreDelete/AtomicPtrMapSharded-12 289ns ± 1%
LoadOrStoreDelete/RWMutexMap-12 342ns ± 2%
LoadOrStoreDelete/SyncMap-12 662ns ± 2%
LoadOrStoreDelete/AtomicPtrMap-12 290ns ± 7%
LoadOrStoreDelete/AtomicPtrMapSharded-12 293ns ± 2%
LookupPositive/RWMutexMap-12 101ns ±26%
LookupPositive/SyncMap-12 202ns ± 2%
LookupPositive/AtomicPtrMap-12 71.1ns ± 2%
LookupPositive/AtomicPtrMapSharded-12 73.2ns ± 1%
LookupNegative/RWMutexMap-12 119ns ± 1%
LookupNegative/SyncMap-12 154ns ± 1%
LookupNegative/AtomicPtrMap-12 84.7ns ± 3%
LookupNegative/AtomicPtrMapSharded-12 86.8ns ± 1%
Concurrent/FixedKeys_1PercentWrites_RWMutexMap-12 1.32µs ± 2%
Concurrent/FixedKeys_1PercentWrites_SyncMap-12 52.7ns ±10%
Concurrent/FixedKeys_1PercentWrites_AtomicPtrMap-12 31.8ns ±20%
Concurrent/FixedKeys_1PercentWrites_AtomicPtrMapSharded-12 24.0ns ±15%
Concurrent/FixedKeys_10PercentWrites_RWMutexMap-12 860ns ± 3%
Concurrent/FixedKeys_10PercentWrites_SyncMap-12 68.8ns ±20%
Concurrent/FixedKeys_10PercentWrites_AtomicPtrMap-12 98.6ns ± 7%
Concurrent/FixedKeys_10PercentWrites_AtomicPtrMapSharded-12 42.0ns ±25%
Concurrent/FixedKeys_50PercentWrites_RWMutexMap-12 1.17µs ± 3%
Concurrent/FixedKeys_50PercentWrites_SyncMap-12 136ns ±34%
Concurrent/FixedKeys_50PercentWrites_AtomicPtrMap-12 286ns ± 3%
Concurrent/FixedKeys_50PercentWrites_AtomicPtrMapSharded-12 115ns ±35%
Concurrent/ChangingKeys_1PercentWrites_RWMutexMap-12 1.27µs ± 2%
Concurrent/ChangingKeys_1PercentWrites_SyncMap-12 5.01µs ± 3%
Concurrent/ChangingKeys_1PercentWrites_AtomicPtrMap-12 38.1ns ± 3%
Concurrent/ChangingKeys_1PercentWrites_AtomicPtrMapSharded-12 22.6ns ± 2%
Concurrent/ChangingKeys_10PercentWrites_RWMutexMap-12 1.08µs ± 2%
Concurrent/ChangingKeys_10PercentWrites_SyncMap-12 5.97µs ± 1%
Concurrent/ChangingKeys_10PercentWrites_AtomicPtrMap-12 390ns ± 2%
Concurrent/ChangingKeys_10PercentWrites_AtomicPtrMapSharded-12 93.6ns ± 1%
Concurrent/ChangingKeys_50PercentWrites_RWMutexMap-12 1.77µs ± 2%
Concurrent/ChangingKeys_50PercentWrites_SyncMap-12 8.07µs ± 2%
Concurrent/ChangingKeys_50PercentWrites_AtomicPtrMap-12 1.61µs ± 2%
Concurrent/ChangingKeys_50PercentWrites_AtomicPtrMapSharded-12 386ns ± 1%
Updates #231
PiperOrigin-RevId: 346614776
|
|
Recursive make is difficult to follow and debug. Drop this by using
internal functions, which, while difficult, are easier than trying to
following recursive invokations.
Further simplify the Makefile by collapsing the image bits and removing
the tools/vm directory, which is effectively unused.
Fixes #4952
PiperOrigin-RevId: 346569133
|
|
PiperOrigin-RevId: 346496532
|
