Age | Commit message (Collapse) | Author |
|
This change makes the checklocks analyzer considerable more powerful, adding:
* The ability to traverse complex structures, e.g. to have multiple nested
fields as part of the annotation.
* The ability to resolve simple anonymous functions and closures, and perform
lock analysis across these invocations. This does not apply to closures that
are passed elsewhere, since it is not possible to know the context in which
they might be invoked.
* The ability to annotate return values in addition to receivers and other
parameters, with the same complex structures noted above.
* Ignoring locking semantics for "fresh" objects, i.e. objects that are
allocated in the local frame (typically a new-style function).
* Sanity checking of locking state across block transitions and returns, to
ensure that no unexpected locks are held.
Note that initially, most of these findings are excluded by a comprehensive
nogo.yaml. The findings that are included are fundamental lock violations.
The changes here should be relatively low risk, minor refactorings to either
include necessary annotations to simplify the code structure (in general
removing closures in favor of methods) so that the analyzer can be easily
track the lock state.
This change additional includes two changes to nogo itself:
* Sanity checking of all types to ensure that the binary and ast-derived
types have a consistent objectpath, to prevent the bug above from occurring
silently (and causing much confusion). This also requires a trick in
order to ensure that serialized facts are consumable downstream. This can
be removed with https://go-review.googlesource.com/c/tools/+/331789 merged.
* A minor refactoring to isolation the objdump settings in its own package.
This was originally used to implement the sanity check above, but this
information is now being passed another way. The minor refactor is preserved
however, since it cleans up the code slightly and is minimal risk.
PiperOrigin-RevId: 382613300
|
|
Update/remove most syserror errors to linuxerr equivalents. For list
of removed errors, see //pkg/syserror/syserror.go.
PiperOrigin-RevId: 382574582
|
|
Update all instances of the above errors to the faster linuxerr implementation.
With the temporary linuxerr.Equals(), no logical changes are made.
PiperOrigin-RevId: 382306655
|
|
Remove three syserror entries duplicated in linuxerr. Because of the
linuxerr.Equals method, this is a mere change of return values from
syserror to linuxerr definitions.
Done with only these three errnos as CLs removing all grow to a significantly
large size.
PiperOrigin-RevId: 382173835
|
|
Add Equals method to compare syserror and unix.Errno errors to linuxerr errors.
This will facilitate removal of syserror definitions in a followup, and
finding needed conversions from unix.Errno to linuxerr.
PiperOrigin-RevId: 380909667
|
|
- Allow the gofer client to use most xattr namespaces. As documented by the
updated comment, this is consistent with e.g. Linux's FUSE client, and allows
gofers to provide extended attributes from FUSE filesystems.
- Make tmpfs' listxattr omit xattrs in the "trusted" namespace for
non-privileged users.
PiperOrigin-RevId: 378778854
|
|
PiperOrigin-RevId: 376932659
|
|
O_PATH is now implemented in vfs2.
Fixes #2782.
PiperOrigin-RevId: 373861410
|
|
PiperOrigin-RevId: 371163405
|
|
PiperOrigin-RevId: 371015541
|
|
PiperOrigin-RevId: 369967629
|
|
PiperOrigin-RevId: 369686285
|
|
While using remote-validation, the vast majority of time spent during
FS operations is re-walking the path to check for modifications and
then closing the file given that in most cases it has not been
modified externally.
This change introduces a new 9P message called MultiGetAttr which bulks
query attributes of several files in one shot. The returned attributes are
then used to update cached dentries before they are walked. File attributes
are updated for files that still exist. Dentries that have been deleted are
removed from the cache. And negative cache entries are removed if a new
file/directory was created externally. Similarly, synthetic dentries are
replaced if a file/directory is created externally.
The bulk update needs to be carefull not to follow symlinks, cross mount
points, because the gofer doesn't know how to resolve symlinks and where
mounts points are located. It also doesn't walk to the parent ("..") to
avoid deadlocks.
Here are the results:
Workload VFS1 VFS2 Change
bazel action 115s 70s 28.8s
Stat/100 11,043us 7,623us 974us
Updates #1638
PiperOrigin-RevId: 369325957
|
|
A skeleton implementation of cgroupfs. It supports trivial cpu and
memory controllers with no support for hierarchies.
PiperOrigin-RevId: 366561126
|
|
Split usermem package to help remove syserror dependency in go_marshal.
New hostarch package contains code not dependent on syserror.
PiperOrigin-RevId: 365651233
|
|
On Linux these are meant to be equivalent to POLLIN/POLLOUT. Rather
than hack these on in sys_poll etc it felt cleaner to just cleanup
the call sites to notify for both events. This is what linux does
as well.
Fixes #5544
PiperOrigin-RevId: 364859977
|
|
PiperOrigin-RevId: 362406813
|
|
This validates that struct fields if annotated with "// checklocks:mu" where
"mu" is a mutex field in the same struct then access to the field is only
done with "mu" locked.
All types that are guarded by a mutex must be annotated with
// +checklocks:<mutex field name>
For more details please refer to README.md.
PiperOrigin-RevId: 360729328
|
|
This removes a three-lock deadlock between fdnotifier.notifier.mu,
epoll.EventPoll.listsMu, and baseEndpoint.mu.
A lock order comment was added to epoll/epoll.go.
Also fix unsafe access of baseEndpoint.connected/receiver.
PiperOrigin-RevId: 358515191
|
|
See https://github.com/golang/go/issues/19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.
PiperOrigin-RevId: 358071574
|
|
PiperOrigin-RevId: 357090170
|
|
PiperOrigin-RevId: 356868412
|
|
Linux does the same thing.
Reported-by: syzbot+6c79385c930c929d1d9e@syzkaller.appspotmail.com
PiperOrigin-RevId: 356854562
|
|
PiperOrigin-RevId: 356645022
|
|
Our implementation of vfs.CheckDeleteSticky was not consistent with Linux,
specifically not consistent with fs/linux.h:check_sticky().
One of the biggest differences was that the vfs implementation did not
allow the owner of the sticky directory to delete files inside it that belonged
to other users.
This change makes our implementation consistent with Linux.
Also adds an integration test to check for this. This bug is also present in
VFS1.
Updates #3027
PiperOrigin-RevId: 355557425
|
|
PiperOrigin-RevId: 354367665
|
|
This also causes inotify events to be generated when reading files for exec.
This change also requires us to adjust splice+inotify tests due to
discrepancies between gVisor and Linux behavior. Note that these discrepancies
existed before; we just did not exercise them previously. See comment for more
details.
Fixes #5348.
PiperOrigin-RevId: 353907187
|
|
IN_CLOSE should only be generated when a file description loses its last
reference; not when a file descriptor is closed.
See fs/file_table.c:__fput.
Updates #5348.
PiperOrigin-RevId: 353810697
|
|
Fixes #5113.
PiperOrigin-RevId: 353313374
|
|
PiperOrigin-RevId: 352904728
|
|
These are primarily simplification and lint mistakes. However, minor
fixes are also included and tests added where appropriate.
PiperOrigin-RevId: 351425971
|
|
AtomicPtrMap is a generic concurrent map from arbitrary keys to arbitrary
pointer values.
Benchmarks:
name time/op
StoreDelete/RWMutexMap-12 335ns ± 1%
StoreDelete/SyncMap-12 705ns ± 3%
StoreDelete/AtomicPtrMap-12 287ns ± 4%
StoreDelete/AtomicPtrMapSharded-12 289ns ± 1%
LoadOrStoreDelete/RWMutexMap-12 342ns ± 2%
LoadOrStoreDelete/SyncMap-12 662ns ± 2%
LoadOrStoreDelete/AtomicPtrMap-12 290ns ± 7%
LoadOrStoreDelete/AtomicPtrMapSharded-12 293ns ± 2%
LookupPositive/RWMutexMap-12 101ns ±26%
LookupPositive/SyncMap-12 202ns ± 2%
LookupPositive/AtomicPtrMap-12 71.1ns ± 2%
LookupPositive/AtomicPtrMapSharded-12 73.2ns ± 1%
LookupNegative/RWMutexMap-12 119ns ± 1%
LookupNegative/SyncMap-12 154ns ± 1%
LookupNegative/AtomicPtrMap-12 84.7ns ± 3%
LookupNegative/AtomicPtrMapSharded-12 86.8ns ± 1%
Concurrent/FixedKeys_1PercentWrites_RWMutexMap-12 1.32µs ± 2%
Concurrent/FixedKeys_1PercentWrites_SyncMap-12 52.7ns ±10%
Concurrent/FixedKeys_1PercentWrites_AtomicPtrMap-12 31.8ns ±20%
Concurrent/FixedKeys_1PercentWrites_AtomicPtrMapSharded-12 24.0ns ±15%
Concurrent/FixedKeys_10PercentWrites_RWMutexMap-12 860ns ± 3%
Concurrent/FixedKeys_10PercentWrites_SyncMap-12 68.8ns ±20%
Concurrent/FixedKeys_10PercentWrites_AtomicPtrMap-12 98.6ns ± 7%
Concurrent/FixedKeys_10PercentWrites_AtomicPtrMapSharded-12 42.0ns ±25%
Concurrent/FixedKeys_50PercentWrites_RWMutexMap-12 1.17µs ± 3%
Concurrent/FixedKeys_50PercentWrites_SyncMap-12 136ns ±34%
Concurrent/FixedKeys_50PercentWrites_AtomicPtrMap-12 286ns ± 3%
Concurrent/FixedKeys_50PercentWrites_AtomicPtrMapSharded-12 115ns ±35%
Concurrent/ChangingKeys_1PercentWrites_RWMutexMap-12 1.27µs ± 2%
Concurrent/ChangingKeys_1PercentWrites_SyncMap-12 5.01µs ± 3%
Concurrent/ChangingKeys_1PercentWrites_AtomicPtrMap-12 38.1ns ± 3%
Concurrent/ChangingKeys_1PercentWrites_AtomicPtrMapSharded-12 22.6ns ± 2%
Concurrent/ChangingKeys_10PercentWrites_RWMutexMap-12 1.08µs ± 2%
Concurrent/ChangingKeys_10PercentWrites_SyncMap-12 5.97µs ± 1%
Concurrent/ChangingKeys_10PercentWrites_AtomicPtrMap-12 390ns ± 2%
Concurrent/ChangingKeys_10PercentWrites_AtomicPtrMapSharded-12 93.6ns ± 1%
Concurrent/ChangingKeys_50PercentWrites_RWMutexMap-12 1.77µs ± 2%
Concurrent/ChangingKeys_50PercentWrites_SyncMap-12 8.07µs ± 2%
Concurrent/ChangingKeys_50PercentWrites_AtomicPtrMap-12 1.61µs ± 2%
Concurrent/ChangingKeys_50PercentWrites_AtomicPtrMapSharded-12 386ns ± 1%
Updates #231
PiperOrigin-RevId: 346614776
|
|
Fixes #4991
PiperOrigin-RevId: 345800333
|
|
These options allow overriding the signal that gets sent to the process when
I/O operations are available on the file descriptor, rather than the default
`SIGIO` signal. Doing so also populates `siginfo` to contain extra information
about which file descriptor caused the event (`si_fd`) and what events happened
on it (`si_band`). The logic around which FD is populated within `si_fd`
matches Linux's, which means it has some weird edge cases where that value may
not actually refer to a file descriptor that is still valid.
This CL also ports extra S/R logic regarding async handler in VFS2.
Without this, async I/O handlers aren't properly re-registered after S/R.
PiperOrigin-RevId: 345436598
|
|
Previous experience has shown that these types of wrappers tends to create two
kinds of problems: hidden allocations (e.g. each call to
FileReadWriteSeeker.Read/Write allocates a usermem.BytesIO on the heap) and
hidden lock ordering problems (e.g. VFS1 splice deadlocks). Since this is only
needed by fsimpl/verity, move it there.
PiperOrigin-RevId: 345377830
|
|
PiperOrigin-RevId: 343196927
|
|
This lets us avoid treating a value of 0 as one reference. All references
using the refsvfs2 template must call InitRefs() before the reference is
incremented/decremented, or else a panic will occur. Therefore, it should be
pretty easy to identify missing InitRef calls during testing.
Updates #1486.
PiperOrigin-RevId: 341411151
|
|
PiperOrigin-RevId: 340536306
|
|
The waits-for relationship between an epoll instance and an inotify fd should
be restored.
This fixes flaky inotify vfs2 tests.
PiperOrigin-RevId: 340531367
|
|
Also refactor the template and CheckedObject interface to make this cleaner.
Updates #1486.
PiperOrigin-RevId: 339577120
|
|
Updates #1486.
PiperOrigin-RevId: 338832085
|
|
Inode number consistency checks are now skipped in save/restore tests for
reasons described in greatest detail in StatTest.StateDoesntChangeAfterRename.
They pass in VFS1 due to the bug described in new test case
SimpleStatTest.DifferentFilesHaveDifferentDeviceInodeNumberPairs.
Fixes #1663
PiperOrigin-RevId: 338776148
|
|
Our current reference leak checker uses finalizers to verify whether an object
has reached zero references before it is garbage collected. There are multiple
problems with this mechanism, so a rewrite is in order.
With finalizers, there is no way to guarantee that a finalizer will run before
the program exits. When an unreachable object with a finalizer is garbage
collected, its finalizer will be added to a queue and run asynchronously. The
best we can do is run garbage collection upon sandbox exit to make sure that
all finalizers are enqueued.
Furthermore, if there is a chain of finalized objects, e.g. A points to B
points to C, garbage collection needs to run multiple times before all of the
finalizers are enqueued. The first GC run will register the finalizer for A but
not free it. It takes another GC run to free A, at which point B's finalizer
can be registered. As a result, we need to run GC as many times as the length
of the longest such chain to have a somewhat reliable leak checker.
Finally, a cyclical chain of structs pointing to one another will never be
garbage collected if a finalizer is set. This is a well-known issue with Go
finalizers (https://github.com/golang/go/issues/7358). Using leak checking on
filesystem objects that produce cycles will not work and even result in memory
leaks.
The new leak checker stores reference counted objects in a global map when
leak check is enabled and removes them once they are destroyed. At sandbox
exit, any remaining objects in the map are considered as leaked. This provides
a deterministic way of detecting leaks without relying on the complexities of
finalizers and garbage collection.
This approach has several benefits over the former, including:
- Always detects leaks of objects that should be destroyed very close to
sandbox exit. The old checker very rarely detected these leaks, because it
relied on garbage collection to be run in a short window of time.
- Panics if we forgot to enable leak check on a ref-counted object (we will try
to remove it from the map when it is destroyed, but it will never have been
added).
- Can store extra logging information in the map values without adding to the
size of the ref count struct itself. With the size of just an int64, the ref
count object remains compact, meaning frequent operations like IncRef/DecRef
are more cache-efficient.
- Can aggregate leak results in a single report after the sandbox exits.
Instead of having warnings littered in the log, which were
non-deterministically triggered by garbage collection, we can print all
warning messages at once. Note that this could also be a limitation--the
sandbox must exit properly for leaks to be detected.
Some basic benchmarking indicates that this change does not significantly
affect performance when leak checking is enabled, which is understandable
since registering/unregistering is only done once for each filesystem object.
Updates #1486.
PiperOrigin-RevId: 338685972
|
|
This fixes reference leaks related to accidentally forgetting to DecRef()
after calling one or the other.
PiperOrigin-RevId: 336918922
|
|
Singleton filesystem like devpts and devtmpfs have a single filesystem shared
among all mounts, so they acquire a "self-reference" when initialized that
must be released when the entire virtual filesystem is released at sandbox
exit.
PiperOrigin-RevId: 336828852
|
|
Merkle tree library was originally using Read/Seek to access data and
tree, since the parameters are io.ReadSeeker. This could cause race
conditions if multiple threads accesses the same fd to read. Here we
change to use ReaderAt, and implement it with PRead to make it thread
safe.
PiperOrigin-RevId: 336779260
|
|
PiperOrigin-RevId: 336694658
|
|
PiperOrigin-RevId: 335077195
|
|
Updates #1663
PiperOrigin-RevId: 333539293
|
|
PiperOrigin-RevId: 333447255
|