| Age | Commit message (Collapse) | Author |
|---|
|
Updates #1035
PiperOrigin-RevId: 293194631
|
|
Updates #1198
Opening host pipes (by spinning in fdpipe) and host sockets is not yet
complete, and will be done in a future CL.
Major differences from VFS1 gofer client (sentry/fs/gofer), with varying levels
of backportability:
- "Cache policies" are replaced by InteropMode, which control the behavior of
timestamps in addition to caching. Under InteropModeExclusive (analogous to
cacheAll) and InteropModeWritethrough (analogous to cacheAllWritethrough),
client timestamps are *not* written back to the server (it is not possible in
9P or Linux for clients to set ctime, so writing back client-authoritative
timestamps results in incoherence between atime/mtime and ctime). Under
InteropModeShared (analogous to cacheRemoteRevalidating), client timestamps
are not used at all (remote filesystem clocks are authoritative). cacheNone
is translated to InteropModeShared + new option
filesystemOptions.specialRegularFiles.
- Under InteropModeShared, "unstable attribute" reloading for permission
checks, lookup, and revalidation are fused, which is feasible in VFS2 since
gofer.filesystem controls path resolution. This results in a ~33% reduction
in RPCs for filesystem operations compared to cacheRemoteRevalidating. For
example, consider stat("/foo/bar/baz") where "/foo/bar/baz" fails
revalidation, resulting in the instantiation of a new dentry:
VFS1 RPCs:
getattr("/") // fs.MountNamespace.FindLink() => fs.Inode.CheckPermission() => gofer.inodeOperations.check() => gofer.inodeOperations.UnstableAttr()
walkgetattr("/", "foo") = fid1 // fs.Dirent.walk() => gofer.session.Revalidate() => gofer.cachePolicy.Revalidate()
clunk(fid1)
getattr("/foo") // CheckPermission
walkgetattr("/foo", "bar") = fid2 // Revalidate
clunk(fid2)
getattr("/foo/bar") // CheckPermission
walkgetattr("/foo/bar", "baz") = fid3 // Revalidate
clunk(fid3)
walkgetattr("/foo/bar", "baz") = fid4 // fs.Dirent.walk() => gofer.inodeOperations.Lookup
getattr("/foo/bar/baz") // linux.stat() => gofer.inodeOperations.UnstableAttr()
VFS2 RPCs:
getattr("/") // gofer.filesystem.walkExistingLocked()
walkgetattr("/", "foo") = fid1 // gofer.filesystem.stepExistingLocked()
clunk(fid1)
// No getattr: walkgetattr already updated metadata for permission check
walkgetattr("/foo", "bar") = fid2
clunk(fid2)
walkgetattr("/foo/bar", "baz") = fid3
// No clunk: fid3 used for new gofer.dentry
// No getattr: walkgetattr already updated metadata for stat()
- gofer.filesystem.unlinkAt() does not require instantiation of a dentry that
represents the file to be deleted. Updates #898.
- gofer.regularFileFD.OnClose() skips Tflushf for regular files under
InteropModeExclusive, as it's nonsensical to request a remote file flush
without flushing locally-buffered writes to that remote file first.
- Symlink targets are cached when InteropModeShared is not in effect.
- p9.QID.Path (which is already required to be unique for each file within a
server, and is accordingly already synthesized from device/inode numbers in
all known gofers) is used as-is for inode numbers, rather than being mapped
along with attr.RDev in the client to yet another synthetic inode number.
- Relevant parts of fsutil.CachingInodeOperations are inlined directly into
gofer package code. This avoids having to duplicate part of its functionality
in fsutil.HostMappable.
PiperOrigin-RevId: 293190213
|
|
Add a file lock implementation that can be embedded into various filesystem
implementations.
Updates #1480
PiperOrigin-RevId: 292614758
|
|
WritableSource is a convenience interface used for files that can
be written to, e.g. /proc/net/ipv4/tpc_sack. It reads max of 4KB
and only from offset 0 which should cover most cases. It can be
extended as neeed.
Updates #1195
PiperOrigin-RevId: 292056924
|
|
PiperOrigin-RevId: 292021389
|
|
Because the abi will depend on the core types for marshalling (usermem,
context, safemem, safecopy), these need to be flattened from the sentry
directory. These packages contain no sentry-specific details.
PiperOrigin-RevId: 291811289
|
|
PiperOrigin-RevId: 291745021
|
|
PiperOrigin-RevId: 291471892
|
|
PiperOrigin-RevId: 291041576
|
|
PiperOrigin-RevId: 291006713
|
|
PiperOrigin-RevId: 290822487
|
|
Some files were missing the last line break.
PiperOrigin-RevId: 290808898
|
|
Updates #1195
PiperOrigin-RevId: 290298266
|
|
Updates #1195
PiperOrigin-RevId: 290285420
|
|
This is similar to 'Truncate' in vfs1.
Updates https://github.com/google/gvisor/issues/1197
PiperOrigin-RevId: 290139140
|
|
Files not implemented require VFSv2 plumbing into the kernel.
Also, cgroup is not implemented yet.
Updates #1195
PiperOrigin-RevId: 290129176
|
|
Except for one under /proc/sys/net/ipv4/tcp_sack.
/proc/pid/* is still incomplete.
Updates #1195
PiperOrigin-RevId: 290120438
|
|
PiperOrigin-RevId: 289962040
|
|
* Rename syncutil to sync.
* Add aliases to sync types.
* Replace existing usage of standard library sync package.
This will make it easier to swap out synchronization primitives. For example,
this will allow us to use primitives from github.com/sasha-s/go-deadlock to
check for lock ordering violations.
Updates #1472
PiperOrigin-RevId: 289033387
|
|
PiperOrigin-RevId: 288799694
|
|
Updates #1195
PiperOrigin-RevId: 288725745
|
|
- Renamed memfs to tmpfs.
- Copied fileRangeSet bits from fs/fsutil/ to fsimpl/tmpfs/
- Changed tmpfs to be backed by filemem instead of byte slice.
- regularFileReadWriter uses a sync.Pool, similar to gofer client.
PiperOrigin-RevId: 288356380
|
|
- Add FileDescriptionOptions.UseDentryMetadata, which reduces the amount of
boilerplate needed for device FDs and the like between filesystems.
- Switch back to having FileDescription.Init() take references on the Mount and
Dentry; otherwise managing refcounts around failed calls to
OpenDeviceSpecialFile() / Device.Open() is tricky.
PiperOrigin-RevId: 287575574
|
|
Updates #1195
PiperOrigin-RevId: 287269106
|
|
Updates #1195
PiperOrigin-RevId: 287227722
|
|
It was calling Dentry.InsertChild with the dentry's mutex
already locked.
Updates #1035
PiperOrigin-RevId: 286962742
|
|
- Make FilesystemImpl methods that operate on parent directories require
!rp.Done() (i.e. there is at least one path component to resolve) as
precondition and postcondition (in cases where they do not finish path
resolution due to mount boundary / absolute symlink), and require that they
do not need to follow the last path component (the file being created /
deleted) as a symlink. Check for these in VFS.
- Add FilesystemImpl.GetParentDentryAt(), which is required to obtain the old
parent directory for VFS.RenameAt(). (Passing the Dentry to be renamed
instead has the wrong semantics if the file named by the old path is a mount
point since the Dentry will be on the wrong Mount.)
- Update memfs to implement these methods correctly (?), including RenameAt.
- Change fspath.Parse() to allow empty paths (to simplify implementation of
AT_EMPTY_PATH).
- Change vfs.PathOperation to take a fspath.Path instead of a raw pathname;
non-test callers will need to fspath.Parse() pathnames themselves anyway in
order to detect absolute paths and select PathOperation.Start accordingly.
PiperOrigin-RevId: 286934941
|
|
PiperOrigin-RevId: 286616668
|
|
PiperOrigin-RevId: 286281274
|
|
PiperOrigin-RevId: 286248378
|
|
PiperOrigin-RevId: 285231002
|
|
The former is needed for vfs.FileDescription to implement
memmap.MappingIdentity, and the latter is needed to implement getcwd(2).
PiperOrigin-RevId: 285051855
|
|
PiperOrigin-RevId: 284892289
|
|
PiperOrigin-RevId: 283610781
|
|
- Remove the Filesystem argument from DentryImpl.*Ref(); in general DentryImpls
that need the Filesystem for reference counting will probably also need it
for other interface methods that don't plumb Filesystem, so it's easier to
just store a pointer to the filesystem in the DentryImpl.
- Add a pointer to the VirtualFilesystem to Filesystem, which is needed by the
gofer client to disown dentries for cache eviction triggered by dentry
reference count changes.
- Rename FilesystemType.NewFilesystem to GetFilesystem; in some cases (e.g.
sysfs, cgroupfs) it's much cleaner for there to be only one Filesystem that
is used by all mounts, and in at least one case (devtmpfs) it's visibly
incorrect not to do so, so NewFilesystem doesn't always actually create and
return a *new* Filesystem.
- Require callers of FileDescription.Init() to increment Mount/Dentry
references. This is because the gofer client may, in the OpenAt() path, take
a reference on a dentry with 0 references, which is safe due to
synchronization that is outside the scope of this CL, and it would be safer
to still have its implementation of DentryImpl.IncRef() check for an
increment for 0 references in other cases.
- Add FileDescription.TryIncRef. This is used by the gofer client to take
references on "special file descriptions" (FDs for files such as pipes,
sockets, and devices), which use per-FD handles (fids) instead of
dentry-shared handles, for sync() and syncfs().
PiperOrigin-RevId: 282473364
|
|
This is required to test filesystems with a non-trivial implementation of
FilesystemImpl.Release(). Propagation isn't handled yet, and umount isn't yet
plumbed out to VirtualFilesystem.UmountAt(), but otherwise the implementation
of umount is believed to be correct.
- Move entering mountTable.seq writer critical sections to callers of
mountTable.{insert,remove}Seqed. This is required since umount(2) must ensure
that no new references are taken on the candidate mount after checking that
it isn't busy, which is only possible by entering a vfs.mountTable.seq writer
critical section before the check and remaining in it until after
VFS.umountRecursiveLocked() is complete. (Linux does the same thing:
fs/namespace.c:do_umount() => lock_mount_hash(),
fs/pnode.c:propagate_mount_busy(), umount_tree(), unlock_mount_hash().)
- It's not possible for dentry deletion to umount while only holding
VFS.mountMu for reading, but it's also very unappealing to hold VFS.mountMu
exclusively around e.g. gofer unlink RPCs. Introduce dentry.mu to avoid these
problems. This means that VFS.mountMu is never acquired for reading, so
change it to a sync.Mutex.
PiperOrigin-RevId: 282444343
|
|
PiperOrigin-RevId: 281795269
|
|
PiperOrigin-RevId: 275650307
|
|
PiperOrigin-RevId: 275139066
|
|
- Pass context.Context to OnClose().
- Pass memmap.MMapOpts to ConfigureMMap() by pointer so that implementations
can actually mutate it as required.
PiperOrigin-RevId: 274934967
|
|
In Linux (include/linux/types.h), mode_t is an unsigned short.
PiperOrigin-RevId: 272956350
|
|
"d_off is the distance from the start of the directory to the start of the next
linux_dirent." - getdents(2).
PiperOrigin-RevId: 270349685
|
|
They are no-ops, so the standard rule works fine.
PiperOrigin-RevId: 268776264
|
|
The simple test script has gotten out of control. Shard this script into
different pieces and attempt to impose order on overall test structure. This
change helps lay some of the foundations for future improvements.
* The runsc/test directories are moved into just test/.
* The runsc/test/testutil package is split into logical pieces.
* The scripts/ directory contains new top-level targets.
* Each test is now responsible for building targets it requires.
* The install functionality is moved into `runsc` itself for simplicity.
* The existing kokoro run_tests.sh file now just calls all (can be split).
After this change is merged, I will create multiple distinct workflows for
Kokoro, one for each of the scripts currently targeted by `run_tests.sh` today,
which should dramatically reduce the time-to-run for the Kokoro tests, and
provides a better foundation for further improvements to the infrastructure.
PiperOrigin-RevId: 267081397
|
|
Migrates all (except 3) seqfile implementations to the vfs.DynamicBytesSource
interface. There should not be any change in functionality due to this migration
itself.
Please note that the following seqfile implementations have not been migrated:
- /proc/filesystems in proc/filesystems.go
- /proc/[pid]/mountinfo in proc/mounts.go
- /proc/[pid]/mounts in proc/mounts.go
This is because these depend on pending changes in /pkg/senty/vfs.
PiperOrigin-RevId: 263880719
|
|
This fixes the implementation ambiguity issues when a filesystem
implementation embeds vfs.DefaultDirectoryFD to its directory FD along
with an internal common fileDescription utility.
For similar reasons also removes FileDescriptionDefaultImpl from
DynamicBytesFileDescriptionImpl.
PiperOrigin-RevId: 263795513
|
|
fsimpl is the keeper of all filesystem implementations in VFS2.
PiperOrigin-RevId: 262617869
|
|
- Unexport Filesystem/Dentry/Inode.
- Support SEEK_CUR in directoryFD.Seek().
- Hold Filesystem.mu before touching directoryFD.off in
directoryFD.Seek().
- Remove deleted Dentries from their parent directory.childLists.
- Remove invalid FIXMEs.
PiperOrigin-RevId: 262400633
|
|
Major differences from the current ("v1") sentry VFS:
- Path resolution is Filesystem-driven (FilesystemImpl methods call
vfs.ResolvingPath methods) rather than VFS-driven (fs package owns a
Dirent tree and calls fs.InodeOperations methods to populate it). This
drastically improves performance, primarily by reducing overhead from
inefficient synchronization and indirection. It also makes it possible
to implement remote filesystem protocols that translate FS system calls
into single RPCs, rather than having to make (at least) one RPC per path
component, significantly reducing the latency of remote filesystems
(especially during cold starts and for uncacheable shared filesystems).
- Mounts are correctly represented as a separate check based on
contextual state (current mount) rather than direct replacement in a
fs.Dirent tree. This makes it possible to support (non-recursive) bind
mounts and mount namespaces.
Included in this CL is fsimpl/memfs, an incomplete in-memory filesystem
that exists primarily to demonstrate intended filesystem implementation
patterns and for benchmarking:
BenchmarkVFS1TmpfsStat/1-6 3000000 497 ns/op
BenchmarkVFS1TmpfsStat/2-6 2000000 676 ns/op
BenchmarkVFS1TmpfsStat/3-6 2000000 904 ns/op
BenchmarkVFS1TmpfsStat/8-6 1000000 1944 ns/op
BenchmarkVFS1TmpfsStat/64-6 100000 14067 ns/op
BenchmarkVFS1TmpfsStat/100-6 50000 21700 ns/op
BenchmarkVFS2MemfsStat/1-6 10000000 197 ns/op
BenchmarkVFS2MemfsStat/2-6 5000000 233 ns/op
BenchmarkVFS2MemfsStat/3-6 5000000 268 ns/op
BenchmarkVFS2MemfsStat/8-6 3000000 477 ns/op
BenchmarkVFS2MemfsStat/64-6 500000 2592 ns/op
BenchmarkVFS2MemfsStat/100-6 300000 4045 ns/op
BenchmarkVFS1TmpfsMountStat/1-6 2000000 679 ns/op
BenchmarkVFS1TmpfsMountStat/2-6 2000000 912 ns/op
BenchmarkVFS1TmpfsMountStat/3-6 1000000 1113 ns/op
BenchmarkVFS1TmpfsMountStat/8-6 1000000 2118 ns/op
BenchmarkVFS1TmpfsMountStat/64-6 100000 14251 ns/op
BenchmarkVFS1TmpfsMountStat/100-6 100000 22397 ns/op
BenchmarkVFS2MemfsMountStat/1-6 5000000 317 ns/op
BenchmarkVFS2MemfsMountStat/2-6 5000000 361 ns/op
BenchmarkVFS2MemfsMountStat/3-6 5000000 387 ns/op
BenchmarkVFS2MemfsMountStat/8-6 3000000 582 ns/op
BenchmarkVFS2MemfsMountStat/64-6 500000 2699 ns/op
BenchmarkVFS2MemfsMountStat/100-6 300000 4133 ns/op
From this we can infer that, on this machine:
- Constant cost for tmpfs stat() is ~160ns in VFS2 and ~280ns in VFS1.
- Per-path-component cost is ~35ns in VFS2 and ~215ns in VFS1, a
difference of about 6x.
- The cost of crossing a mount boundary is about 80ns in VFS2
(MemfsMountStat/1 does approximately the same amount of work as
MemfsStat/2, except that it also crosses a mount boundary). This is an
inescapable cost of the separate mount lookup needed to support bind
mounts and mount namespaces.
PiperOrigin-RevId: 258853946
|
