summaryrefslogtreecommitdiffhomepage
path: root/tools
AgeCommit message (Collapse)Author
2020-10-23Rewrite reference leak checker without finalizers.Dean Deng
Our current reference leak checker uses finalizers to verify whether an object has reached zero references before it is garbage collected. There are multiple problems with this mechanism, so a rewrite is in order. With finalizers, there is no way to guarantee that a finalizer will run before the program exits. When an unreachable object with a finalizer is garbage collected, its finalizer will be added to a queue and run asynchronously. The best we can do is run garbage collection upon sandbox exit to make sure that all finalizers are enqueued. Furthermore, if there is a chain of finalized objects, e.g. A points to B points to C, garbage collection needs to run multiple times before all of the finalizers are enqueued. The first GC run will register the finalizer for A but not free it. It takes another GC run to free A, at which point B's finalizer can be registered. As a result, we need to run GC as many times as the length of the longest such chain to have a somewhat reliable leak checker. Finally, a cyclical chain of structs pointing to one another will never be garbage collected if a finalizer is set. This is a well-known issue with Go finalizers (https://github.com/golang/go/issues/7358). Using leak checking on filesystem objects that produce cycles will not work and even result in memory leaks. The new leak checker stores reference counted objects in a global map when leak check is enabled and removes them once they are destroyed. At sandbox exit, any remaining objects in the map are considered as leaked. This provides a deterministic way of detecting leaks without relying on the complexities of finalizers and garbage collection. This approach has several benefits over the former, including: - Always detects leaks of objects that should be destroyed very close to sandbox exit. The old checker very rarely detected these leaks, because it relied on garbage collection to be run in a short window of time. - Panics if we forgot to enable leak check on a ref-counted object (we will try to remove it from the map when it is destroyed, but it will never have been added). - Can store extra logging information in the map values without adding to the size of the ref count struct itself. With the size of just an int64, the ref count object remains compact, meaning frequent operations like IncRef/DecRef are more cache-efficient. - Can aggregate leak results in a single report after the sandbox exits. Instead of having warnings littered in the log, which were non-deterministically triggered by garbage collection, we can print all warning messages at once. Note that this could also be a limitation--the sandbox must exit properly for leaks to be detected. Some basic benchmarking indicates that this change does not significantly affect performance when leak checking is enabled, which is understandable since registering/unregistering is only done once for each filesystem object. Updates #1486. PiperOrigin-RevId: 338685972
2020-10-20Fix errors when the tagging GitHub releasesTing-Yu Wang
When the commit description contains "commit ", it will be wrongly identified as commit hash. This commit changes to take only lines begins with "commit " as a fix, since the description is always indented by `git log`. Copybara uses merge commit for external contributors, this causes that not all commits contain a Piper ID. Adding `--first-parent` to `git log` so that it only lists commits that contain a Piper ID. PiperOrigin-RevId: 338183812
2020-10-20Fix nogo test in //pkg/tcpip/...Ting-Yu Wang
PiperOrigin-RevId: 338168977
2020-10-19Remove legacy bazel configurations.Adin Scannell
Using the newer bazel rules necessitates a transition from proto1 to proto2. In order to resolve the incompatibility between proto2 and gogoproto, the cri runtimeoptions proto must be vendored. Further, some of the semantics of bazel caching changed during the transition. It is now necessary to: - Ensure that :gopath depends only on pure library targets, as the propagation of go_binary build attributes (pure, static) will affected the generated files (though content remains the same, there are conflicts with respect to the gopath). - Update bazel.mk to include the possibility of binaries in the bazel-out directory, as it will now put runsc and others there. This required some refinements to the mechanism of extracting paths, since some the existing regex resulted in false positives. - Change nogo rules to prevent escape generation on binary targets. For some reason, the newer version of bazel attempted to run the nogo analysis on the binary targets, which fails due to the fact that objdump does not work on the final binary. This must be due to a change in the semantics of aspects in bazel3. PiperOrigin-RevId: 337958324
2020-10-16Refactor shared starlark files.Adin Scannell
PiperOrigin-RevId: 337581114
2020-10-16Refactor nogo to better support ARM.Adin Scannell
PiperOrigin-RevId: 337544107
2020-10-16Cache errors when processing stdlib with nogo.Adin Scannell
PiperOrigin-RevId: 337515664
2020-10-09Add parsers golang benchmarks.Zach Koopmans
Add parser and formatting for golang benchmarks for docker benchmarks. Change adds a library for printing and parsing Test parameters and metrics. Benchmarks use the library to print parameters in the Benchmark title (e.g. the name field in b.Run()), and to report CustomMetrics. Parser uses the library to parse printed data from benchmark output and put it into BigQuery structs. PiperOrigin-RevId: 336365628
2020-10-09Minor nogo restructuring.Adin Scannell
PiperOrigin-RevId: 336343819
2020-10-09Infer receiver name for stateify.Adin Scannell
PiperOrigin-RevId: 336340035
2020-10-08Minor nogo cleanup.Adin Scannell
PiperOrigin-RevId: 336126583
2020-10-07Add staticcheck and staticstyle analyzers.Adin Scannell
This change also adds support to go_stateify for detecting an appropriate receiver name, avoiding a large number of false positives. PiperOrigin-RevId: 335994587
2020-10-07Internal change.gVisor bot
PiperOrigin-RevId: 335960488
2020-10-07Fix text processing in bazel build command.Dean Deng
The extraction of the build target was overfitted before, making build_cmd fail in some environments. PiperOrigin-RevId: 335916651
2020-10-05Internal change.gVisor bot
PiperOrigin-RevId: 335429072
2020-10-02tools/checkescape: trim the "(SB)" before searching it in the allowed listAndrei Vagin
Signed-off-by: Andrei Vagin <avagin@gmail.com>
2020-10-02Save addresses for "allowed" functions.Adin Scannell
PiperOrigin-RevId: 335086850
2020-09-29go-marshal tests should respect build tags.Rahat Mahmood
Previously, the go-marshal-generated tests did not respect build tags. This can cause the test to unbuildable under some build configurations, as the original types the tests refer to may not be defined. This CL copies the build tags from the input files to the test, similar to the generated library; however test packages have an additional constraint. A test package cannot be totally empty (i.e. have no test/example/benchmark defined), otherwise the go compiler returns an error. To ensure the generated test package always contains a testable entity under all build configurations, we now emit an extra test file with no build tags that contains a single no-op example. PiperOrigin-RevId: 334496821
2020-09-29Add nogo check annotations to GitHub.Adin Scannell
When nogo checks are violated, they will automatically posted as annotations on the specific GitHub commit. This allows us to ensure analysis & style rules and have them called out. PiperOrigin-RevId: 334447285
2020-09-29Support embedded fields in go-marshal.Rahat Mahmood
PiperOrigin-RevId: 334437990
2020-09-23make: specify /dev/null for the tail toolAndrei Vagin
2020-09-23Internal change.gVisor bot
PiperOrigin-RevId: 333287864
2020-09-17Provide testing container with docker config file.Ayush Ranjan
This is needed by test/e2e/integration_test:TestCheckpointRestore to check for filesystem versioning. PiperOrigin-RevId: 332285566
2020-09-16Rename marshal.Task to marshal.CopyContext.Rahat Mahmood
CopyContext is a better name for the interface because from go-marshal's perspective, the interface has nothing to do with a task. A kernel.Task happens to implement the interface, but so can other things like MemoryManager and IO sequences. PiperOrigin-RevId: 331959678
2020-09-15Enable automated marshalling for the syscall package.Rahat Mahmood
PiperOrigin-RevId: 331940975
2020-09-11Move the 'marshal' and 'primitive' packages to the 'pkg' directory.Rahat Mahmood
PiperOrigin-RevId: 331256608
2020-09-11Make nogo more robust to variety of stdlib layouts.Michael Pratt
PiperOrigin-RevId: 331206424
2020-09-07Fix make_apt script.Ayush Ranjan
This change makes the following fixes: - When creating a test repo.key, create a secret keyring as other workflows also use secret keyrings only. - We should not be using both --keyring and --secret-keyring options. Just use --secret-keyring. - Pass homedir to all gpg commands. dpkg-sig takes an arg -g which stands for gpgopts. So we need to pass the homedir there too. PiperOrigin-RevId: 330443280
2020-09-03Fix the release workflow.Ayush Ranjan
PiperOrigin-RevId: 330049242
2020-08-31Change nogo failures to test failures, instead of build failures.Adin Scannell
PiperOrigin-RevId: 329408633
2020-08-26Support stdlib analyzers with nogo.Adin Scannell
This immediately revealed an escape analysis violation (!), where the sync.Map was being used in a context that escapes were not allowed. This is a relatively minor fix and is included. PiperOrigin-RevId: 328611237
2020-08-25Provide --secret-keyring parameter (for newer gpg).Adin Scannell
PiperOrigin-RevId: 328403914
2020-08-25Add nogo support to go_binary and go_test targets.Adin Scannell
Updates #3374 PiperOrigin-RevId: 328378700
2020-08-25[go-marshal] Support marshalling for structs with names starting with W.Ayush Ranjan
Due to how marshallable interface implementation was generated, we could not marshal a struct whose named started with W because there was a naming collision with parameter (w io.Writer) and type (w *StuctName). Used "writer" as parameter name to avoid collision. PiperOrigin-RevId: 328343930
2020-08-14Handle URLs better in issue reviverIan Lewis
- Handle urls ending in / - Add some url parsing tests PiperOrigin-RevId: 326750183
2020-08-07Merge pull request #3520 from boyuan-he:marshal-implgVisor bot
PiperOrigin-RevId: 325546935
2020-08-07Remove old benchmark tools.Zach Koopmans
Remove the old benchmark-tools directory, including imports in the WORKSPACE file and associated bazel rules. The new Golang benchmark-tools can be found at //test/benchmarks and it is functionally equivalent, excepting syscall_test which can be found in //test/perf/linux. PiperOrigin-RevId: 325529075
2020-08-07add stub go marshallable implementationBoyuan He
2020-08-06Add bzl_library rules for .bzl files without one.Adin Scannell
PiperOrigin-RevId: 325280924
2020-08-04Enable "make packetimpact-tests" to work.Adin Scannell
This required minor fixes to the bazel wrapper. The "dut_platform" is also changed to "native" to line-up with the system call tests and remove the hard-coded "linux" and "netstack" strings.
2020-08-03Include shim binaries in the Go branch.Adin Scannell
PiperOrigin-RevId: 324615016
2020-07-30Allocate a pseudo-tty for exec.Adin Scannell
Otherwise Ctrl-C will kill the 'docker exec' as opposed to killing the bazel command being run inside the container. PiperOrigin-RevId: 324079339
2020-07-30Drop complex awk step.Adin Scannell
PiperOrigin-RevId: 324023425
2020-07-27Merge pull request #2797 from Rajpratik71:optimization/pip-no-cachegVisor bot
PiperOrigin-RevId: 323508910
2020-07-27Fix for gvisor-builder image.Bhasker Hariharan
As it happens gvisor/tools/bazel.mk:88 useradd --uid $(UID) --non-unique --no-create-home \ adds the user-id to /var/log/lastlog which happens to be a sparse file except Go's tar support can't handle sparse files so it actually tries to allocate the file to seek to the end causing the VM to run out of disk space. See: https://github.com/moby/moby/issues/5419#issuecomment-193876183 The fix is to pass -l to useradd to prevent it from trying to add to lastlog. Fixes #3397 PiperOrigin-RevId: 323492591
2020-07-27Enable RBE for standard-tests.Adin Scannell
PiperOrigin-RevId: 323454998
2020-07-27Clean-up bazel wrapper.Adin Scannell
The bazel server was being started as the wrong user, leading to issues where the container would suddenly exit during a build. We can also simplify the waiting logic by starting the container in two separate steps: those that must complete first, then the asynchronous bit. PiperOrigin-RevId: 323391161
2020-07-25optimize size and time using "--no-cache-dir"Pratik raj
Using "--no-cache-dir" flag in pip install ,make sure dowloaded packages by pip don't cached on system . This is a best practise which make sure to fetch ftom repo instead of using local cached one . Further , in case of Docker Containers , by restricing caching , we can reduce image size. In term of stats , it depends upon the number of python packages multiplied by their respective size . e.g for heavy packages with a lot of dependencies it reduce a lot by don't caching pip packages. Further , more detail information can be found at https://medium.com/sciforce/strategies-of-docker-images-optimization-2ca9cc5719b6
2020-07-24Merge pull request #3356 from amscanne:generics_testsgVisor bot
PiperOrigin-RevId: 323066414
2020-07-24[go-marshal] Update APIAyush Ranjan
- All Marshal* and Unmarshal* methods now require buffers to be correctly sized - Only the Copy{In/Out} variants can handle smaller buffers (or address spaces) PiperOrigin-RevId: 322953881
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-28 18:18:13 +0000