1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
|
# Containing a Real Vulnerability
In the previous two posts we talked about gVisor's
[security design principles](https://gvisor.dev/blog/2019/11/18/gvisor-security-basics-part-1/)
as well as how those are applied in the
[context of networking](https://gvisor.dev/blog/2020/04/02/gvisor-networking-security/).
Recently, a new container escape vulnerability
([CVE-2020-14386](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14386))
was announced that ties these topics well together. gVisor is
[not vulnerable](https://seclists.org/oss-sec/2020/q3/168) to this specific
issue, but it provides an interesting case study to continue our exploration of
gVisor's security. While gVisor is not immune to vulnerabilities,
[we take several steps](https://gvisor.dev/security/) to minimize the impact and
remediate if a vulnerability is found.
## Escaping the Container
First, let’s describe how the discovered vulnerability works. There are numerous
ways one can send and receive bytes over the network with Linux. One of the most
performant ways is to use a ring buffer, which is a memory region shared by the
application and the kernel. These rings are created by calling
[setsockopt(2)](https://man7.org/linux/man-pages/man2/setsockopt.2.html) with
[`PACKET_RX_RING`](https://man7.org/linux/man-pages/man7/packet.7.html) for
receiving and
[`PACKET_TX_RING`](https://man7.org/linux/man-pages/man7/packet.7.html) for
sending packets.
The vulnerability is in the code that reads packets when `PACKET_RX_RING` is
enabled. There is another option
([`PACKET_RESERVE`](https://man7.org/linux/man-pages/man7/packet.7.html)) that
asks the kernel to leave some space in the ring buffer before each packet for
anything the application needs, e.g. control structures. When a packet is
received, the kernel calculates where to copy the packet to, taking the amount
reserved before each packet into consideration. If the amount reserved is large,
the kernel performed an incorrect calculation which could cause an overflow
leading to an out-of-bounds write of up to 10 bytes, controlled by the attacker.
The data in the write is easily controlled using the loopback to send a crafted
packet and receiving it using a `PACKET_RX_RING` with a carefully selected
`PACKET_RESERVE` size.
```c
static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
struct packet_type *pt, struct net_device *orig_dev)
{
// ...
if (sk->sk_type == SOCK_DGRAM) {
macoff = netoff = TPACKET_ALIGN(po->tp_hdrlen) + 16 +
po->tp_reserve;
} else {
unsigned int maclen = skb_network_offset(skb);
// tp_reserve is unsigned int, netoff is unsigned short. Addition can overflow netoff
netoff = TPACKET_ALIGN(po->tp_hdrlen +
(maclen < 16 ? 16 : maclen)) +
po->tp_reserve;
if (po->has_vnet_hdr) {
netoff += sizeof(struct virtio_net_hdr);
do_vnet = true;
}
// Attacker controls netoff and can make macoff be smaller than sizeof(struct virtio_net_hdr)
macoff = netoff - maclen;
}
// ...
// "macoff - sizeof(struct virtio_net_hdr)" can be negative, resulting in a pointer before h.raw
if (do_vnet &&
virtio_net_hdr_from_skb(skb, h.raw + macoff -
sizeof(struct virtio_net_hdr),
vio_le(), true, 0)) {
// ...
```
The [`CAP_NET_RAW`](https://man7.org/linux/man-pages/man7/capabilities.7.html)
capability is required to create the socket above. However, in order to support
common debugging tools like `ping` and `tcpdump`, Docker containers, including
those created for Kubernetes, are given `CAP_NET_RAW` by default and thus may be
able to trigger this vulnerability to elevate privileges and escape the
container.
Next, we are going to explore why this vulnerability doesn’t work in gVisor, and
how gVisor could prevent the escape even if a similar vulnerability existed
inside gVisor’s kernel.
## Default Protections
gVisor does not implement `PACKET_RX_RING`, but **does** support raw sockets
which are required for `PACKET_RX_RING`. Raw sockets are a controversial feature
to support in a sandbox environment. While it allows great customizations for
essential tools like `ping`, it may allow packets to be written to the network
without any validation. In general, allowing an untrusted application to write
crafted packets to the network is a questionable idea and a historical source of
vulnerabilities. With that in mind, if `CAP_NET_RAW` is enabled by default, it
would not be _secure by default_ to run untrusted applications.
After multiple discussions when raw sockets were first implemented, we decided
to disable raw sockets by default, **even if `CAP_NET_RAW` is given to the
application**. Instead, enabling raw sockets in gVisor requires the admin to set
`--net-raw` flag to runsc when configuring the runtime, in addition to requiring
the `CAP_NET_RAW` capability in the application. It comes at the expense that
some tools may not work out of the box, but as part of our
[secure-by-default](https://gvisor.dev/blog/2019/11/18/gvisor-security-basics-part-1/#secure-by-default)
principle, we felt that it was important for the “less secure” configuration to
be explicit.
Since this bug was due to an overflow in the specific Linux implementation of
the packet ring, gVisor's raw socket implementation is not affected. However, if
there were a vulnerability in gVisor, containers would not be allowed to exploit
it by default.
As an alternative way to implement this same constraint, Kubernetes allows
[admission controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)
to be configured to customize requests. Cloud providers can use this to
implement more stringent policies. For example, GKE implements an admission
controller for gVisor that
[removes `CAP_NET_RAW` from gVisor pods](https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods#capabilities)
unless it has been explicitly set in the pod spec.
## Isolated Kernel
gVisor has its own application kernel, called the Sentry, that is distinct from
the host kernel. Just like what you would expect from a kernel, gVisor has a
memory management subsystem, virtual file system, and a full network stack. The
host network is only used as a transport to carry packets in and out the
sandbox[^1]. The loopback interface which is used in the exploit stays
completely inside the sandbox, never reaching the host.
Therefore, even if the Sentry was vulnerable to the attack, there would be two
factors that would prevent a container escape from happening. First, the
vulnerability would be limited to the Sentry, and the attacker would compromise
only the application kernel, bound by a restricted set of
[seccomp](https://en.wikipedia.org/wiki/Seccomp) filters, discussed more in
depth below. Second, the Sentry is a distinct implementation of the API, written
in Go, which provides bounds checking that would have likely prevented access
past the bounds of the shared region (e.g. see
[aio](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/syscalls/linux/vfs2/aio.go;l=210;drc=a11061d78a58ed75b10606d1a770b035ed944b66?q=file:aio&ss=gvisor%2Fgvisor)
or
[kcov](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/kernel/kcov.go;l=272?q=file:kcov&ss=gvisor%2Fgvisor),
which have similar shared regions).
Here, Kubernetes warrants slightly more explanation. gVisor makes pods the unit
of isolation and a pod can run multiple containers. In other words, each pod is
a gVisor instance, and each container is a set of processes running inside
gVisor, isolated via Sentry-internal namespaces like regular containers inside a
pod. If there were a vulnerability in gVisor, the privilege escalation would
allow a container inside the pod to break out to other **containers inside the
same pod**, but the container still **cannot break out of the pod**.
## Defense in Depth
gVisor follows a
[common security principle used at Google](https://cloud.google.com/security/infrastructure/design/resources/google_infrastructure_whitepaper_fa.pdf)
that the system should have two layers of protection, and those layers should
require different compromises to be broken. We apply this principle by assuming
that the Sentry (first layer of defense)
[will be compromised and should not be trusted](https://gvisor.dev/blog/2019/11/18/gvisor-security-basics-part-1/#defense-in-depth).
In order to protect the host kernel from a compromised Sentry, we wrap it around
many security and isolations features to ensure only the minimal set of
functionality from the host kernel is exposed.
First, the sandbox runs inside a cgroup that can limit and throttle host
resources being used. Second, the sandbox joins empty namespaces, including user
and mount, to further isolate from the host. Next, it changes the process root
to a read-only directory that contains only `/proc` and nothing else. Then, it
executes with the unprivileged user/group
[`nobody`](https://en.wikipedia.org/wiki/Nobody_\(username\)) with all
capabilities stripped. Last and most importantly, a seccomp filter is added to
tightly restrict what parts of the Linux syscall surface that gVisor is allowed
to access. The allowed host surface is a far smaller set of syscalls than the
Sentry implements for applications to use. Not only restricting the syscall
being called, but also checking that arguments to these syscalls are within the
expected set. Dangerous syscalls like <code>execve(2)</code>,
<code>open(2)</code>, and <code>socket(2)</code> are prohibited, thus an
attacker isn’t able to execute binaries or acquire new resources on the host.
if there were a vulnerability in gVisor that allowed an attacker to execute code
inside the Sentry, the attacker still has extremely limited privileges on the
host. In fact, a compromised Sentry is much more restricted than a
non-compromised regular container. For CVE-2020-14386 in particular, the attack
would be blocked by more than one security layer: non-privileged user, no
capability, and seccomp filters.
Although the surface is drastically reduced, there is still a chance that there
is a vulnerability in one of the allowed syscalls. That’s why it’s important to
keep the surface small and carefully consider what syscalls are allowed. You can
find the full set of allowed syscalls
[here](https://cs.opensource.google/gvisor/gvisor/+/master:runsc/boot/filter/).
Another possible attack vector is resources that are present in the Sentry, like
open file descriptors. The Sentry has file descriptors that an attacker could
potentially use, such as log files, platform files (e.g. `/dev/kvm`), an RPC
endpoint that allows external communication with the Sentry, and a Netstack
endpoint that connects the sandbox to the network. The Netstack endpoint in
particular is a concern because it gives direct access to the network. It’s an
`AF_PACKET` socket that allows arbitrary L2 packets to be written to the
network. In the normal case, Netstack assembles packets that go out the network,
giving the container control over only the payload. But if the Sentry is
compromised, an attacker can craft packets to the network. In many ways this is
similar to anyone sending random packets over the internet, but still this is a
place where the host kernel surface exposed is larger than we would like it to
be.
## Conclusion
Security comes with many tradeoffs that are often hard to make, such as the
decision to disable raw sockets by default. However, these tradeoffs have served
us well, and we've found them to have paid off over time. CVE-2020-14386 offers
great insight into how multiple layers of protection can be effective against
such an attack.
We cannot guarantee that a container escape will never happen in gVisor, but we
do our best to make it as hard as we possibly can.
If you have not tried gVisor yet, it’s easier than you think. Just follow the
steps in the
[Quick Start](https://gvisor.dev/docs/user_guide/quick_start/docker/) guide.
<br>
<br>
--------------------------------------------------------------------------------
[^1]: Those packets are eventually handled by the host, as it needs to route
them to local containers or send them out the NIC. The packet will be
handled by many switches, routers, proxies, servers, etc. along the way,
which may be subject to their own vulnerabilities.
|
