gvisor Documentation

File: README.md

What is gVisor?

gVisor provides a strong layer of isolation between running applications and
the host operating system. It is an application kernel that implements a
Linux-like interface. Unlike Linux, it is written in a memory-safe
language (Go) and runs in userspace.

gVisor includes an Open Container Initiative (OCI) runtime called runsc
that makes it easy to work with existing container tooling. The runsc runtime
integrates with Docker and Kubernetes, making it simple to run sandboxed
containers.

What isn't gVisor?

gVisor is not a syscall filter (e.g. seccomp-bpf), nor a wrapper over
Linux isolation primitives (e.g. firejail, AppArmor, etc.).
gVisor is also not a VM in the everyday sense of the term (e.g.
VirtualBox, QEMU).

gVisor takes a distinct third approach, providing many security benefits of
VMs while maintaining the lower resource footprint, fast startup, and
flexibility of regular userspace applications.

Why does gVisor exist?

Containers are not a sandbox. While containers have
revolutionized how we develop, package, and deploy applications, using them to
run untrusted or potentially malicious code without additional isolation is not
a good idea. While using a single, shared kernel allows for efficiency and
performance gains, it also means that container escape is possible with a single
vulnerability.

gVisor is an application kernel for containers. It limits the host kernel
surface accessible to the application while still giving the application access
to all the features it expects. Unlike most kernels, gVisor does not assume or
require a fixed set of physical resources; instead, it leverages existing host
kernel functionality and runs as a normal process. In other words, gVisor
implements Linux by way of Linux.

gVisor should not be confused with technologies and tools to harden containers
against external threats, provide additional integrity checks, or limit the
scope of access for a service. One should always be careful about what data is
made available to a container.

Documentation

User documentation and technical architecture, including quick start guides, can
be found at gvisor.dev.

Installing from source

gVisor builds on x86_64 and ARM64. Other architectures may become available in
the future.

For the purposes of these instructions, bazel and other build
dependencies are wrapped in a build container. It is possible to use
bazel directly, or type make help for standard targets.

Requirements

Make sure the following dependencies are installed:

Linux 5.6+
Docker version 17.09.0 or greater

Building

Build a release tarball containing runsc, the containerd-shim-runsc-v1
containerd shim, and a few sidecar binaries that runsc expects to find in a
gvisor-bin/ directory next to itself, then extract it to /usr/local/bin:

make release-tarball DESTINATION=bin/
sudo tar -C /usr/local/bin -xf bin/gvisor.tar.bz2

To build specific libraries or binaries, you can specify the target:

make build TARGETS="//pkg/tcpip:tcpip"

Building directly with Bazel (without Docker)

Using Bazel directly isn't recommended due to the extra overhead, but in order
to get started:

Look at the build dockerfile for the canonical
list of needed dependencies.
Install and use bazelisk. Otherwise, make sure your bazel
version matches the one listed in the .bazelversion file.

After setting up dependencies, using Bazel is similar to the Makefile:

bazel build -c opt //debian:gvisor-release-tar

Testing

To run standard test suites, you can use:

make unit-tests
make tests

To run specific tests, you can specify the target:

# Makefile
make test TARGETS="//runsc:version_test"
# Bazel
bazel test //runsc:version_test

Mac OS

Some packages support running tests directly on macOS. At the time of this
writing, gVisor requires bazel 8, which you can install via homebrew:

brew install bazel@8

# You can then run the tests, e.g.:
$(brew --prefix bazel@8)/bin/bazel test --macos_sdk_version=$(xcrun --show-sdk-version) -- //tools/nogo/... //tools/check{aligned,const,escape,linkname,locks,unsafe}/...

Using `go get`

This project uses bazel to build and manage dependencies. A synthetic
go branch is maintained that is compatible with standard go tooling for
convenience. This is useful for external packages and libraries that depend on
gVisor subpackages (e.g. userspace networking via Netstack) to import gVisor Go
code into their Go projects.

Select this branch explicitly with the go branch query. @latest resolves
master, which requires Bazel and is not compatible with standard Go tooling:

go get gvisor.dev/gvisor/pkg/tcpip/transport/tcp@go

NOTE: runsc builds from this branch are not supported. gVisor and
runsc require several binaries (some of which are not even written in Go) in
order to function. The go branch is supported in a best effort capacity, and
direct development on this branch is not supported. Development should occur on
the master branch, which is then reflected into the go branch.

Community & Governance

See GOVERNANCE.md for project governance information.

The gvisor-users mailing list and
gvisor-dev mailing list are good starting points for
questions and discussion.

Security Policy

See SECURITY.md.

Contributing

See Contributing.md.

File: examples/seccheck/README.md

gVisor Seccheck Examples

C++ Server

This directory provides an example of a monitoring process that receives
connections from gVisor sandboxes and prints the traces to stdout. The example
contains two main files:

server.cc: this is where main() and all the code is. It sets up a
server listening to a Unix-domain socket located at
/tmp/gvisor_events.sock or a configurable location via a command line
argument.
pod_init.json: this file contains the trace configuration that should be
passed to runsc. It can be done either via --pod-init-config flag or
using runsc trace create command. Note that the socket location is
specified in this file, in case you change it.

Usage

First start the server in a terminal, which will block and wait for new
connections:

shell

$ bazel run examples/seccheck:server_cc
Socket address /tmp/gvisor_events.sock

Now, open a second, separate terminal to trigger a simple example using
runsc do. We recommend building and executing runsc directly from the source
tree via Bazel to ensure it contains the latest trace points configuration
needed for this example, rather than accidentally using an older system-provided
binary:

shell

$ bazel run runsc -- \
    --rootless --network=none \
    --pod-init-config=$PWD/examples/seccheck/pod_init.json \
    do echo 123

Back at the server terminal, you can see the following traces being output:

text

Connection accepted
Start => id:     "runsc-329739" cwd: "/home/fvoznika" args: "echo" args: "123"
E Open sysno:    257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libc.so.6" flags: 524288
X Open exit      { errorno: 2 } sysno: 257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libc.so.6" flags: 524288
E Open sysno:    257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libc.so.6" flags: 524288
X Open exit      { errorno: 2 } sysno: 257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libc.so.6" flags: 524288
...
TaskExit =>
Connection closed

Connection messages indicate when runsc connected to and disconnected from the
server. Then there is a trace for container start and a few syscalls to
open(2) for searching libraries. You can change pod_init.json to configure
the trace session to your liking.

To set this up with Docker, you can add the --pod-init-config flag when the
runtime is installed:

shell

# Rebuild runsc and copy to a stable location
$ mkdir -p bin
$ make copy TARGETS=//:release DESTINATION=bin/
$ sudo cp -r --preserve=mode bin/runsc bin/containerd-shim-runsc-v1 bin/gvisor-bin /usr/local/bin/
$ sudo /usr/local/bin/runsc install --runtime=runsc-trace -- \
    --pod-init-config=$PWD/examples/seccheck/pod_init.json

# Restart the Docker daemon so it reloads its configuration and discovers
# the new "runsc-trace" runtime we just installed.
$ sudo systemctl restart docker

# Run a test container to trigger trace events and verify it works!
$ docker run --rm --runtime=runsc-trace hello-world

Go Server Implementation

If you prefer to write your seccheck server in Go instead of C++, you can use
the reference Go server implementation located at
pkg/sentry/seccheck/sinks/remote/server/server.go.
It provides a robust, native sink to receive and process trace payloads.

Learn More

To learn more about how seccheck works under the hood—including how to
discover trace points, configure advanced sessions dynamically, and understand
the core architecture—please refer to the full
Seccheck Documentation.

File: g3doc/README.md

What is gVisor?

gVisor can be used with Docker, Kubernetes, or directly using runsc. Use the
links below to see detailed instructions for each of them:

Docker: The quickest and easiest way
to get started.
Kubernetes: Isolate Pods in your
K8s cluster with gVisor.
OCI Quick Start: Expert mode. Customize
gVisor for your environment.

What does gVisor do?

gVisor provides a virtualized environment in order to sandbox containers. The
system interfaces normally implemented by the host kernel are moved into a
distinct, per-sandbox application kernel in order to minimize the risk of a
container escape exploit. gVisor does not introduce large fixed overheads
however, and still retains a process-like model with respect to resource
utilization.

How is this different?

Two other approaches are commonly taken to provide stronger isolation than
native containers.

gVisor is not a syscall filter (e.g. seccomp-bpf), nor a wrapper over
Linux isolation primitives (e.g. firejail, AppArmor, etc.).
gVisor is also not a VM in the everyday sense of the term (e.g.
VirtualBox, QEMU).

gVisor takes a distinct third approach, providing many security benefits of
VMs while maintaining the lower resource footprint, fast startup, and
flexibility of regular userspace applications.

Let's take a closer look.

Machine-level virtualization, such as KVM and Xen, exposes
virtualized hardware to a guest kernel via a Virtual Machine Monitor (VMM). This
virtualized hardware is generally enlightened (paravirtualized) and additional
mechanisms can be used to improve the visibility between the guest and host
(e.g. balloon drivers, paravirtualized spinlocks). Running containers in
distinct virtual machines can provide great isolation, compatibility and
performance (though nested virtualization may bring challenges in this area),
but for containers it often requires additional proxies and agents, and may
require a larger resource footprint and slower start-up times.

Rule-based execution, such as seccomp, SELinux and
AppArmor, allows the specification of a fine-grained security policy
for an application or container. These schemes typically rely on hooks
implemented inside the host kernel to enforce the rules. If the surface can be
made small enough, then this is an excellent way to sandbox applications and
maintain native performance. However, in practice it can be extremely difficult
(if not impossible) to reliably define a policy for arbitrary, previously
unknown applications, making this approach challenging to apply universally.

Rule-based execution is often combined with additional layers for
defense-in-depth.

gVisor provides a third isolation mechanism, distinct from those above.

gVisor intercepts application system calls and acts as the guest kernel, without
the need for translation through virtualized hardware. gVisor may be thought of
as either a merged guest kernel and VMM, or as seccomp on steroids. This
architecture allows it to provide a flexible resource footprint (i.e. one based
on threads and memory mappings, not fixed guest physical resources) while also
lowering the fixed costs of virtualization. However, this comes at the price of
reduced application compatibility and higher per-system call overhead.

On top of this, gVisor employs rule-based execution to provide defense-in-depth
(details below).

gVisor's approach is similar to User Mode Linux (UML), although UML
virtualizes hardware internally and thus provides a fixed resource footprint.

Each of the above approaches may excel in distinct scenarios. For example,
machine-level virtualization will face challenges achieving high density, while
gVisor may provide poor performance for system call heavy workloads.

Why Go?

gVisor is written in Go in order to avoid security pitfalls that can
plague kernels. With Go, there are strong types, built-in bounds checks, no
uninitialized variables, no use-after-free, no stack overflow, and a built-in
race detector. However, the use of Go has its challenges, and the runtime often
introduces performance overhead.

What are the different components?

A gVisor sandbox consists of multiple processes. These processes collectively
comprise an environment in which one or more containers can be run.

Each sandbox has its own isolated instance of:

The Sentry, which is a kernel that runs the containers and intercepts
and responds to system calls made by the application.

Each container running in the sandbox has its own isolated instance of:

A Gofer which provides file system access to the containers.

What is runsc?

The entrypoint to running a sandboxed container is the runsc executable.
runsc implements the Open Container Initiative (OCI) runtime
specification, which is used by Docker and Kubernetes. This means that OCI
compatible filesystem bundles can be run by runsc. Filesystem bundles are
comprised of a config.json file containing container configuration, and a root
filesystem for the container. Please see the OCI runtime spec
for more information on filesystem bundles. runsc implements multiple commands
that perform various functions such as starting, stopping, listing, and querying
the status of containers.

Sentry {#sentry}

The Sentry is the largest component of gVisor. It can be thought of as a
application kernel. The Sentry implements all the kernel functionality needed by
the application, including: system calls, signal delivery, memory management and
page faulting logic, the threading model, and more.

When the application makes a system call, the
Platform redirects the call to the Sentry,
which will do the necessary work to service it. It is important to note that the
Sentry does not pass system calls through to the host kernel. As a userspace
application, the Sentry will make some host system calls to support its
operation, but it does not allow the application to directly control the system
calls it makes. For example, the Sentry is not able to open files directly; file
system operations that extend beyond the sandbox (not internal /proc files,
pipes, etc) are sent to the Gofer, described below.

Gofer {#gofer}

The Gofer is a standard host process which is started with each container and
communicates with the Sentry via the 9P protocol over a socket or shared
memory channel. The Sentry process is started in a restricted seccomp container
without access to file system resources. The Gofer mediates all access to these
resources, providing an additional level of isolation.

Application {#application}

The application is a normal Linux binary provided to gVisor in an OCI runtime
bundle. gVisor aims to provide an environment equivalent to Linux, so
applications should be able to run unmodified. However, gVisor does not
presently implement every system call, /proc file, or /sys file so some
incompatibilities may occur. See Compatibility
for more information.

File: images/agent/README.md

Build Agent

This is the image used by the build agent. It is built and bundled via a
separate packaging mechanism in order to provide local caching and to ensure
that there is better build provenance. Note that continuous integration system
will generally deploy new agents from the primary branch, and will only deploy
as instances are recycled. Updates to this image should be made carefully.

File: images/syzkaller/README.md

syzkaller is an unsupervised coverage-guided kernel fuzzer.

How to run syzkaller.

First, we need to load a syzkaller docker image:

bash

make load-syzkaller

or we can rebuild it to use an up-to-date version of the master branch:

bash

make rebuild-syzkaller

Then we need to create a directory with all artifacts that we will need to run a
syzkaller. Then we will bind-mount this directory to a docker container.

We need to build runsc and place it on the artifact directory:

bash

make RUNTIME_DIR=/tmp/syzkaller refresh

The next step is to create a syzkaller config. We can copy the default one and
customize it:

bash

cp images/syzkaller/default-gvisor-config.cfg /tmp/syzkaller/syzkaller.cfg

Now we can start syzkaller in a docker container:

bash

docker run --privileged -it --rm \
    -v /tmp/syzkaller:/tmp/syzkaller \
    gvisor.dev/images/syzkaller:latest

All logs will be in /tmp/syzkaller/workdir.

How to run a syz repro.

We need to repeat all preparation steps from the previous section and save a
syzkaller repro in /tmp/syzkaller/repro.

Now we can run syz-repro to reproduce a crash:

bash

docker run --privileged -it --rm -v \
    /tmp/syzkaller:/tmp/syzkaller --entrypoint="" \
    gvisor.dev/images/syzkaller:latest ./bin/syz-repro -config \
    /tmp/syzkaller/syzkaller.cfg /tmp/syzkaller/repro

File: images/README.md

Container Images

This directory contains all images used by tests.

Note that all these images must be pushed to the testing project hosted on
[Google Container Registry][gcr]. This will happen automatically as part of
continuous integration. This will speed up loading as images will not need to be
built from scratch for each test run.

Image tooling is accessible via make, specifically via tools/images.mk.

Why make?

Make is used because it can bootstrap the default image, which contains
bazel and all other parts of the toolchain.

Listing images

To list all images, use make list-all-images from the top-level directory.

Loading and referencing images

To build a specific image, use make load- from the top-level directory.
This will ensure that an image gvisor.dev/images/:latest is available.

Images should always be referred to via the gvisor.dev/images canonical path.
This tag exists only locally, but serves to decouple tests from the underlying
image infrastructure.

The continuous integration system can either take fine-grained dependencies on
single images via individual load targets, or pull all images via a single
load-all-test-images invocation.

Adding new images

To add a new image, create a new directory under images containing a
Dockerfile and any other files that the image requires. You may choose to add to
an existing subdirectory if applicable, or create a new one.

All images will be tagged and memoized using a hash of the directory contents.
As a result, every image should be made completely reproducible if possible.
This means using fixed tags and fixed versions whenever feasible.

Note that images should also be made architecture-independent if possible. The
build scripts will handle loading the appropriate architecture onto the machine
and tagging it with the single canonical tag.

Add a load- dependency in the Makefile if the image is required for a
particular set of tests. This target will pull the tag from the image repository
if available.

Building and pushing images

All images can be built manually by running build- and pushed using
push-. Note that you can also use push-all-images. Note that pushing
will require appropriate permissions in the project.

The continuous integration system can either take fine-grained dependencies on
individual push targets, or ensure all images are up-to-date with a single
push-all-images invocation.

Multi-Arch images

By default, the image is built for host architecture. Cross-building can be
achieved by specifying ARCH variable to make. For example:

text

$ make ARCH=aarch64 rebuild-default

File: pkg/usermem/README.md

This package defines primitives for sentry access to application memory.

Major types:

The IO interface represents a virtual address space and provides I/O
methods on that address space. IO is the lowest-level primitive. The
primary implementation of the IO interface is mm.MemoryManager.
IOSequence represents a collection of individually-contiguous address
ranges in a IO that is operated on sequentially, analogous to Linux's
struct iov_iter.

Major usage patterns:

Access to a task's virtual memory, subject to the application's memory
protections and while running on that task's goroutine, from a context that
is at or above the level of the kernel package (e.g. most syscall
implementations in syscalls/linux); use the kernel.Task.Copy* wrappers
defined in kernel/task_usermem.go.
Access to a task's virtual memory, from a context that is at or above the
level of the kernel package, but where any of the above constraints does
not hold (e.g. PTRACE_POKEDATA, which ignores application memory
protections); obtain the task's mm.MemoryManager by calling
kernel.Task.MemoryManager, and call its IO methods directly.
Access to a task's virtual memory, from a context that is below the level of
the kernel package (e.g. filesystem I/O); clients must pass I/O arguments
from higher layers, usually in the form of an IOSequence. The
kernel.Task.SingleIOSequence and kernel.Task.IovecsIOSequence functions
in kernel/task_usermem.go are convenience functions for doing so.

File: pkg/sync/README.md

sync

This package provides additional synchronization primitives not provided by the
Go stdlib 'sync' package. It is partially derived from the upstream 'sync'
package from go1.10.

File: pkg/state/README.md

State Encoding and Decoding

The state package implements the encoding and decoding of data structures for
go_stateify. This package is designed for use cases other than the standard
encoding packages, e.g. gob and json. Principally:

This package operates on complex object graphs and accurately serializes and
restores all relationships. That is, you can have things like: intrusive
pointers, cycles, and pointer chains of arbitrary depths. These are not
handled appropriately by existing encoders. This is not an implementation
flaw: the formats themselves are not capable of representing these graphs,
as they can only generate directed trees.
This package allows installing order-dependent load callbacks and then
resolves that graph at load time, with cycle detection. Similarly, there is
no analogous feature possible in the standard encoders.
This package handles the resolution of interfaces, based on a registered
type name. For interface objects type information is saved in the serialized
format. This is generally true for gob as well, but it works differently.

Here's an overview of how encoding and decoding works.

Encoding

Encoding produces a statefile, which contains a list of chunks of the form
(header, payload). The payload can either be some raw data, or a series of
encoded wire objects representing some object graph. All encoded objects are
defined in the wire subpackage.

Encoding of an object graph begins with encodeState.Save.

1. Memory Map & Encoding

To discover relationships between potentially interdependent data structures
(for example, a struct may contain pointers to members of other data
structures), the encoder first walks the object graph and constructs a memory
map of the objects in the input graph. As this walk progresses, objects are
queued in the pending list and items are placed on the deferred list as they
are discovered. No single object will be encoded multiple times, but the
discovered relationships between objects may change as more parts of the overall
object graph are discovered.

The encoder starts at the root object and recursively visits all reachable
objects, recording the address ranges containing the underlying data for each
object. This is stored as a segment set (addrSet), mapping address ranges to
the of the object occupying the range; see encodeState.values. Note that there
is special handling for zero-sized types and map objects during this process.

Additionally, the encoder assigns each object a unique identifier which is used
to indicate relationships between objects in the statefile; see objectID in
encode.go.

2. Type Serialization

The enoder will subsequently serialize all information about discovered types,
including field names. These are used during decoding to reconcile these types
with other internally registered types.

3. Object Serialization

With a full address map, and all objects correctly encoded, all object encodings
are serialized. The assigned objectIDs aren't explicitly encoded in the
statefile. The order of object messages in the stream determine their IDs.

Example

Given the following data structure definitions:

type system struct {
    o *outer
    i *inner
}

type outer struct {
    a  int64
    cn *container
}

type container struct {
    n    uint64
    elem *inner
}

type inner struct {
    c    container
    x, y uint64
}

Initialized like this:

o := outer{
    a: 10,
    cn: nil,
}
i := inner{
    x: 20,
    y: 30,
    c: container{},
}
s := system{
    o: &o,
    i: &i,
}

o.cn = &i.c
o.cn.elem = &i

Encoding will produce an object stream like this:

text

g0r1 = struct{
     i: g0r3,
     o: g0r2,
}
g0r2 = struct{
     a: 10,
     cn: g0r3.c,
}
g0r3 = struct{
     c: struct{
             elem: g0r3,
             n: 0u,
     },
     x: 20u,
     y: 30u,
}

Note how g0r3.c is correctly encoded as the underlying container object for
inner.c, and how the pointer from outer.cn points to it, despite system.i
being discovered after the pointer to it in system.o.cn. Also note that
decoding isn't strictly reliant on the order of encoded object stream, as long
as the relationship between objects are correctly encoded.

Decoding

Decoding reads the statefile and reconstructs the object graph. Decoding begins
in decodeState.Load. Decoding is performed in a single pass over the object
stream in the statefile, and a subsequent pass over all deserialized objects is
done to fire off all loading callbacks in the correctly defined order. Note that
introducing cycles is possible here, but these are detected and an error will be
returned.

Decoding is relatively straight forward. For most primitive values, the decoder
constructs an appropriate object and fills it with the values encoded in the
statefile. Pointers need special handling, as they must point to a value
allocated elsewhere. When values are constructed, the decoder indexes them by
their objectIDs in decodeState.objectsByID. The target of pointers are
resolved by searching for the target in this index by their objectID; see
decodeState.register. For pointers to values inside another value (fields in a
pointer, elements of an array), the decoder uses the accessor path to walk to
the appropriate location; see walkChild.

File: pkg/sentry/fsutil/README.md

This package provides utilities for implementing virtual filesystem objects.

[TOC]

Page cache

CachingInodeOperations implements a page cache for files that cannot use the
host page cache. Normally these are files that store their data in a remote
filesystem. This also applies to files that are accessed on a platform that does
not support directly memory mapping host file descriptors (e.g. the ptrace
platform).

An CachingInodeOperations buffers regions of a single file into memory. It is
owned by an fs.Inode, the in-memory representation of a file (all open file
descriptors are backed by an fs.Inode). The fs.Inode provides operations for
reading memory into an CachingInodeOperations, to represent the contents of
the file in-memory, and for writing memory out, to relieve memory pressure on
the kernel and to synchronize in-memory changes to filesystems.

An CachingInodeOperations enables readable and/or writable memory access to
file content. Files can be mapped shared or private, see mmap(2). When a file is
mapped shared, changes to the file via write(2) and truncate(2) are reflected in
the shared memory region. Conversely, when the shared memory region is modified,
changes to the file are visible via read(2). Multiple shared mappings of the
same file are coherent with each other. This is consistent with Linux.

When a file is mapped private, updates to the mapped memory are not visible to
other memory mappings. Updates to the mapped memory are also not reflected in
the file content as seen by read(2). If the file is changed after a private
mapping is created, for instance by write(2), the change to the file may or may
not be reflected in the private mapping. This is consistent with Linux.

An CachingInodeOperations keeps track of ranges of memory that were modified
(or "dirtied"). When the file is explicitly synced via fsync(2), only the dirty
ranges are written out to the filesystem. Any error returned indicates a failure
to write all dirty memory of an CachingInodeOperations to the filesystem. In
this case the filesystem may be in an inconsistent state. The same operation can
be performed on the shared memory itself using msync(2). If neither fsync(2) nor
msync(2) is performed, then the dirty memory is written out in accordance with
the CachingInodeOperations eviction strategy (see below) and there is no
guarantee that memory will be written out successfully in full.

Memory allocation and eviction

An CachingInodeOperations implements the following allocation and eviction
strategy:

Memory is allocated and brought up to date with the contents of a file when
a region of mapped memory is accessed (or "faulted on").
Dirty memory is written out to filesystems when an fsync(2) or msync(2)
operation is performed on a memory mapped file, for all memory mapped files
when saved, and/or when there are no longer any memory mappings of a range
of a file, see munmap(2). As the latter implies, in the absence of a panic
or SIGKILL, dirty memory is written out for all memory mapped files when an
application exits.
Memory is freed when there are no longer any memory mappings of a range of a
file (e.g. when an application exits). This behavior is consistent with
Linux for shared memory that has been locked via mlock(2).

Notably, memory is not allocated for read(2) or write(2) operations. This means
that reads and writes to the file are only accelerated by an
CachingInodeOperations if the file being read or written has been memory
mapped and if the shared memory has been accessed at the region being read or
written. This diverges from Linux which buffers memory into a page cache on
read(2) proactively (i.e. readahead) and delays writing it out to filesystems on
write(2) (i.e. writeback). The absence of these optimizations is not visible to
applications beyond less than optimal performance when repeatedly reading and/or
writing to same region of a file. See Future Work for plans to
implement these optimizations.

Additionally, memory held by CachingInodeOperationss is currently unbounded in
size. An CachingInodeOperations does not write out dirty memory and free it
under system memory pressure. This can cause pathological memory usage.

When memory is written back, an CachingInodeOperations may write regions of
shared memory that were never modified. This is due to the strategy of
minimizing page faults (see below) and handling only a subset of memory write
faults. In the absence of an application or sentry crash, it is guaranteed that
if a region of shared memory was written to, it is written back to a filesystem.

Life of a shared memory mapping

A file is memory mapped via mmap(2). For example, if A is an address, an
application may execute:

text

mmap(A, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

This creates a shared mapping of fd that reflects 4k of the contents of fd
starting at offset 0, accessible at address A. This in turn creates a virtual
memory area region ("vma") which indicates that [A, A+0x1000) is now a valid
address range for this application to access.

At this point, memory has not been allocated in the file's
CachingInodeOperations. It is also the case that the address range [A,
A+0x1000) has not been mapped on the host on behalf of the application. If the
application then tries to modify 8 bytes of the shared memory:

text

char buffer[] = "aaaaaaaa";
memcpy(A, buffer, 8);

The host then sends a SIGSEGV to the sentry because the address range [A,
A+8) is not mapped on the host. The SIGSEGV indicates that the memory was
accessed writable. The sentry looks up the vma associated with [A, A+8),
finds the file that was mapped and its CachingInodeOperations. It then calls
CachingInodeOperations.Translate which allocates memory to back [A, A+8).
It may choose to allocate more memory (i.e. do "readahead") to minimize
subsequent faults.

Memory that is allocated comes from a host tmpfs file (see
pgalloc.MemoryFile). The host tmpfs file memory is brought up to date with the
contents of the mapped file on its filesystem. The region of the host tmpfs file
that reflects the mapped file is then mapped into the host address space of the
application so that subsequent memory accesses do not repeatedly generate a
SIGSEGV.

The range that was allocated, including any extra memory allocation to minimize
faults, is marked dirty due to the write fault. This overcounts dirty memory if
the extra memory allocated is never modified.

To make the scenario more interesting, imagine that this application spawns
another process and maps the same file in the exact same way:

text

mmap(A, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

Imagine that this process then tries to modify the file again but with only 4
bytes:

text

char buffer[] = "bbbb";
memcpy(A, buffer, 4);

Since the first process has already mapped and accessed the same region of the
file writable, CachingInodeOperations.Translate is called but returns the
memory that has already been allocated rather than allocating new memory. The
address range [A, A+0x1000) reflects the same cached view of the file as the
first process sees. For example, reading 8 bytes from the file from either
process via read(2) starting at offset 0 returns a consistent "bbbbaaaa".

When this process no longer needs the shared memory, it may do:

text

munmap(A, 0x1000);

At this point, the modified memory cached by the CachingInodeOperations is not
written back to the file because it is still in use by the first process that
mapped it. When the first process also does:

text

munmap(A, 0x1000);

Then the last memory mapping of the file at the range [0, 0x1000) is gone. The
file's CachingInodeOperations then starts writing back memory marked dirty to
the file on its filesystem. Once writing completes, regardless of whether it was
successful, the CachingInodeOperations frees the memory cached at the range
[0, 0x1000).

Subsequent read(2) or write(2) operations on the file go directly to the
filesystem since there no longer exists memory for it in its
CachingInodeOperations.

Future Work

Page cache

The sentry does not yet implement the readahead and writeback optimizations for
read(2) and write(2) respectively. To do so, on read(2) and/or write(2) the
sentry must ensure that memory is allocated in a page cache to read or write
into. However, the sentry cannot boundlessly allocate memory. If it did, the
host would eventually OOM-kill the sentry+application process. This means that
the sentry must implement a page cache memory allocation strategy that is
bounded by a global user or container imposed limit. When this limit is
approached, the sentry must decide from which page cache memory should be freed
so that it can allocate more memory. If it makes a poor decision, the sentry may
end up freeing and re-allocating memory to back regions of files that are
frequently used, nullifying the optimization (and in some cases causing worse
performance due to the overhead of memory allocation and general management).
This is a form of "cache thrashing".

In Linux, much research has been done to select and implement a lightweight but
optimal page cache eviction algorithm. Linux makes use of hardware page bits to
keep track of whether memory has been accessed. The sentry does not have direct
access to hardware. Implementing a similarly lightweight and optimal page cache
eviction algorithm will need to either introduce a kernel interface to obtain
these page bits or find a suitable alternative proxy for access events.

In Linux, readahead happens by default but is not always ideal. For instance,
for files that are not read sequentially, it would be more ideal to simply read
from only those regions of the file rather than to optimistically cache some
number of bytes ahead of the read (up to 2MB in Linux) if the bytes cached won't
be accessed. Linux implements the fadvise64(2) system call for applications to
specify that a range of a file will not be accessed sequentially. The advice bit
FADV_RANDOM turns off the readahead optimization for the given range in the
given file. However fadvise64 is rarely used by applications so Linux implements
a readahead backoff strategy if reads are not sequential. To ensure that
application performance is not degraded, the sentry must implement a similar
backoff strategy.

Docker & Self-Hosted Deployment Verified OCI

File: README.md Copy Chapter

What is gVisor? Copy Chapter

What **isn't** gVisor? Copy Chapter

Why does gVisor exist? Copy Chapter

Documentation Copy Chapter

Installing from source Copy Chapter

Requirements Copy Section

Building Copy Section

Building directly with Bazel (without Docker) Copy Section

Testing Copy Section

Mac OS Copy Section

Using `go get` Copy Section

Community & Governance Copy Chapter

Security Policy Copy Chapter

Contributing Copy Chapter

File: examples/seccheck/README.md Copy Chapter

gVisor Seccheck Examples

C++ Server Copy Chapter

Usage Copy Section

Go Server Implementation Copy Chapter

Learn More Copy Chapter

File: g3doc/README.md Copy Chapter

What is gVisor?

What does gVisor do? Copy Chapter

How is this different? Copy Chapter

Why Go? Copy Chapter

What are the different components? Copy Chapter

What is runsc? Copy Chapter

Sentry {#sentry} Copy Section

Gofer {#gofer} Copy Section

Application {#application} Copy Section

File: images/agent/README.md Copy Chapter

Build Agent

File: images/syzkaller/README.md Copy Chapter

How to run syzkaller.

How to run a syz repro.

File: images/README.md Copy Chapter

Container Images

Why make? Copy Chapter

Listing images Copy Chapter

Loading and referencing images Copy Chapter

Adding new images Copy Chapter

Building and pushing images Copy Chapter

Multi-Arch images Copy Chapter

File: pkg/usermem/README.md Copy Chapter

File: pkg/sync/README.md Copy Chapter

sync

File: pkg/state/README.md Copy Chapter

State Encoding and Decoding

Encoding Copy Chapter

1. Memory Map & Encoding Copy Section

2. Type Serialization Copy Section

3. Object Serialization Copy Section

Example Copy Section

Decoding Copy Chapter

File: pkg/sentry/fsutil/README.md Copy Chapter

Page cache Copy Chapter

Memory allocation and eviction Copy Section

Life of a shared memory mapping Copy Section

Future Work Copy Chapter

Page cache Copy Section

Explore Related Technologies & Topics

File: README.md

What is gVisor?

What isn't gVisor?

Why does gVisor exist?

Documentation

Installing from source

Requirements

Building

Building directly with Bazel (without Docker)

Testing

Mac OS

Using `go get`

Community & Governance

Security Policy

Contributing

File: examples/seccheck/README.md

C++ Server

Usage

Go Server Implementation

Learn More

File: g3doc/README.md

What does gVisor do?

How is this different?

Why Go?

What are the different components?

What is runsc?

Sentry {#sentry}

Gofer {#gofer}

Application {#application}

File: images/agent/README.md

File: images/syzkaller/README.md

File: images/README.md

Why make?

Listing images

Loading and referencing images

Adding new images

Building and pushing images

Multi-Arch images

File: pkg/usermem/README.md

File: pkg/sync/README.md

File: pkg/state/README.md

Encoding

1. Memory Map & Encoding

2. Type Serialization

3. Object Serialization

Example

Decoding

File: pkg/sentry/fsutil/README.md

Page cache

Memory allocation and eviction

Life of a shared memory mapping

Future Work

Page cache