1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
# Inotify
Inotify implements the like-named filesystem event notification system for the
sentry, see `inotify(7)`.
## Architecture
For the most part, the sentry implementation of inotify mirrors the Linux
architecture. Inotify instances (i.e. the fd returned by inotify_init(2)) are
backed by a pseudo-filesystem. Events are generated from various places in the
sentry, including the [syscall layer][syscall_dir], the [vfs layer][dirent] and
the [process fd table][fd_map]. Watches are stored in inodes and generated
events are queued to the inotify instance owning the watches for delivery to the
user.
## Objects
Here is a brief description of the existing and new objects involved in the
sentry inotify mechanism, and how they interact:
### [`fs.Inotify`][inotify]
- An inotify instances, created by inotify_init(2)/inotify_init1(2).
- The inotify fd has a `fs.Dirent`, supports filesystem syscalls to read
events.
- Has multiple `fs.Watch`es, with at most one watch per target inode, per
inotify instance.
- Has an instance `id` which is globally unique. This is *not* the fd number
for this instance, since the fd can be duped. This `id` is not externally
visible.
### [`fs.Watch`][watch]
- An inotify watch, created/deleted by
inotify_add_watch(2)/inotify_rm_watch(2).
- Owned by an `fs.Inotify` instance, each watch keeps a pointer to the
`owner`.
- Associated with a single `fs.Inode`, which is the watch `target`. While the
watch is active, it indirectly pins `target` to memory. See the "Reference
Model" section for a detailed explanation.
- Filesystem operations on `target` generate `fs.Event`s.
### [`fs.Event`][event]
- A simple struct encapsulating all the fields for an inotify event.
- Generated by `fs.Watch`es and forwarded to the watches' `owner`s.
- Serialized to the user during read(2) syscalls on the associated
`fs.Inotify`'s fd.
### [`fs.Dirent`][dirent]
- Many inotify events are generated inside dirent methods. Events are
generated in the dirent methods rather than `fs.Inode` methods because some
events carry the name of the subject node, and node names are generally
unavailable in an `fs.Inode`.
- Dirents do not directly contain state for any watches. Instead, they forward
notifications to the underlying `fs.Inode`.
### [`fs.Inode`][inode]
- Interacts with inotify through `fs.Watch`es.
- Inodes contain a map of all active `fs.Watch`es on them.
- An `fs.Inotify` instance can have at most one `fs.Watch` per inode.
`fs.Watch`es on an inode are indexed by their `owner`'s `id`.
- All inotify logic is encapsulated in the [`Watches`][inode_watches] struct
in an inode. Logically, `Watches` is the set of inotify watches on the
inode.
## Reference Model
The sentry inotify implementation has a complex reference model. An inotify
watch observes a single inode. For efficient lookup, the state for a watch is
stored directly on the target inode. This state needs to be persistent for the
lifetime of watch. Unlike usual filesystem metadata, the watch state has no
"on-disk" representation, so they cannot be reconstructed by the filesystem if
the inode is flushed from memory. This effectively means we need to keep any
inodes with actives watches pinned to memory.
We can't just hold an extra ref on the inode to pin it to memory because some
filesystems (such as gofer-based filesystems) don't have persistent inodes. In
such a filesystem, if we just pin the inode, nothing prevents the enclosing
dirent from being GCed. Once the dirent is GCed, the pinned inode is
unreachable -- these filesystems generate a new inode by re-reading the node
state on the next walk. Incidentally, hardlinks also don't work on these
filesystems for this reason.
To prevent the above scenario, when a new watch is added on an inode, we *pin*
the dirent we used to reach the inode. Note that due to hardlinks, this dirent
may not be the only dirent pointing to the inode. Attempting to set an inotify
watch via multiple hardlinks to the same file results in the same watch being
returned for both links. However, for each new dirent we use to reach the same
inode, we add a new pin. We need a new pin for each new dirent used to reach the
inode because we have no guarantees about the deletion order of the different
links to the inode.
## Lock Ordering
There are 4 locks related to the inotify implementation:
- `Inotify.mu`: the inotify instance lock.
- `Inotify.evMu`: the inotify event queue lock.
- `Watch.mu`: the watch lock, used to protect pins.
- `fs.Watches.mu`: the inode watch set mu, used to protect the collection of
watches on the inode.
The correct lock ordering for inotify code is:
`Inotify.mu` -> `fs.Watches.mu` -> `Watch.mu` -> `Inotify.evMu`.
We need a distinct lock for the event queue because by the time a goroutine
attempts to queue a new event, it is already holding `fs.Watches.mu`. If we used
`Inotify.mu` to also protect the event queue, this would violate the above lock
ordering.
[dirent]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/fs/dirent.go
[event]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/fs/inotify_event.go
[fd_map]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/kernel/fd_map.go
[inode]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/fs/inode.go
[inode_watches]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/fs/inode_inotify.go
[inotify]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/fs/inotify.go
[syscall_dir]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/syscalls/linux/
[watch]: https://gvisor.googlesource.com/gvisor/+/master/pkg/sentry/fs/inotify_watch.go
|
