## File: README.md # youki: A container runtime in Rust [](https://discord.gg/zHnyXKSQFD) [](https://github.com/youki-dev/youki/graphs/contributors) [](https://github.com/youki-dev/youki/actions) [](https://app.fossa.com/projects/git%2Bgithub.com%2Fyouki-dev%2Fyouki?ref=badge_shield) **youki** is an implementation of the [OCI runtime-spec](https://github.com/opencontainers/runtime-spec) in Rust, similar to [runc](https://github.com/opencontainers/runc). Your ideas are welcome [here](https://github.com/youki-dev/youki/issues/10). # ๐Ÿท๏ธ About the name youki is pronounced as /joสŠki/ or yoh-key. youki is named after the Japanese word 'youki', which means 'a container'. In Japanese language, youki also means 'cheerful', 'merry', or 'hilarious'. # ๐Ÿš€ Quick Start > [!TIP] > You can immediately set up your environment with youki on GitHub Codespaces and try it out. > > [](https://codespaces.new/youki-dev/youki?quickstart=1) > ```console > just build > docker run --runtime youki hello-world > sudo podman run --cgroup-manager=cgroupfs --runtime /workspaces/youki/youki hello-world > ``` [User Documentation](https://youki-dev.github.io/youki/user/basic_setup.html#quick-install) # ๐ŸŽฏ Motivation Here is why we are writing a new container runtime in Rust. - Rust is one of the best languages to implement the oci-runtime spec. Many very nice container tools are currently written in Go. However, the container runtime requires the use of system calls, which requires a bit of special handling when implemented in Go. This is tricky (e.g. _namespaces(7)_, _fork(2)_); with Rust too, but it's not that tricky. And, unlike in C, Rust provides the benefit of memory safety. While Rust is not yet a major player in the container field, it has the potential to contribute a lot: something this project attempts to exemplify. - youki has the potential to be faster and use less memory than runc, and therefore work in environments with tight memory usage requirements. Here is a simple benchmark of a container from creation to deletion. | Runtime | Time (mean ยฑ ฯƒ) | Range (min โ€ฆ max) | vs youki(mean) | Version | | -------- | -------- | -------- | -------- | -------- | | youki | 111.5 ms ยฑ 11.6 ms | 84.0 ms ยฑ 142.5 ms | 100% | 0.3.3 | | runc | 224.6 ms ยฑ 12.0 ms | 190.5 ms ยฑ 255.4 ms | 200% | 1.1.7 | | crun | 47.3 ms ยฑ 2.8 ms | 42.4 ms ยฑ 56.2 ms | 42% | 1.15 | Details about the benchmark - A command used for the benchmark ```bash hyperfine --prepare 'sudo sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' --warmup 10 --min-runs 100 'sudo ./youki create -b tutorial a && sudo ./youki start a && sudo ./youki delete -f a' ``` - Environment ```console $ ./youki info Version 0.3.3 Commit 4f3c8307 Kernel-Release 6.5.0-35-generic Kernel-Version #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 Architecture x86_64 Operating System Ubuntu 22.04.4 LTS Cores 16 Total Memory 63870 Cgroup setup unified Cgroup mounts Namespaces enabled mount enabled uts enabled ipc enabled user enabled pid enabled network enabled cgroup enabled Capabilities CAP_BPF available CAP_PERFMON available CAP_CHECKPOINT_RESTORE available ``` - I have fun implementing this. In fact, this may be the most important. # ๐Ÿ“ Status of youki **youki** has aced real-world use cases, including containerd's e2e test, and is now adopted by several production environments. See [Adopters and Use Cases](https://youki-dev.github.io/youki/community/adopters_and_use_cases.html) for public examples. We have [our roadmap](https://github.com/orgs/containers/projects/15). # ๐Ÿ”— Related project - [youki-dev/oci-spec-rs](https://github.com/youki-dev/oci-spec-rs) - OCI Runtime and Image Spec in Rust # ๐ŸŽจ Design and implementation of youki The User and Developer Documentation for youki is hosted at [https://youki-dev.github.io/youki/](https://youki-dev.github.io/youki/) # ๐ŸŽฌ Getting Started Local build is only supported on Linux. For other platforms, please use the [Vagrantfile](#setting-up-vagrant) that we have prepared. You can also spin up a fully preconfigured development environment in the cloud with [GitHub Codespaces](https://docs.github.com/en/codespaces/getting-started/quickstart). ## Requires - Rust(See [here](https://www.rust-lang.org/tools/install)), edition 2024 - linux kernel โ‰ฅ 5.3 ## Dependencies To install `just`, follow the instruction [here](https://github.com/casey/just#installation). ### Debian, Ubuntu and related distributions ```console sudo apt-get install \ pkg-config \ libsystemd-dev \ build-essential \ libelf-dev \ libseccomp-dev \ libclang-dev \ libssl-dev ``` ### Fedora, CentOS, RHEL and related distributions ```console sudo dnf install \ pkg-config \ systemd-devel \ elfutils-libelf-devel \ libseccomp-devel \ clang-devel \ openssl-devel ``` ## Build ```bash git clone git@github.com:youki-dev/youki.git cd youki just youki-dev # or youki-release ./youki -h # you can get information about youki command ``` ## Tutorial ### Requires - Docker(See [here](https://docs.docker.com/engine/install)) ### Create and run a container Let's try to run a container that executes `sleep 30` with youki. This tutorial may need root permission. ```bash git clone git@github.com:youki-dev/youki.git cd youki just youki-dev # or youki-release mkdir -p tutorial/rootfs cd tutorial # use docker to export busybox into the rootfs directory docker export $(docker create busybox) | tar -C rootfs -xvf - ``` Then, we need to prepare a configuration file. This file contains metadata and specs for a container, such as the process to run, environment variables to inject, sandboxing features to use, etc. ```bash ../youki spec # will generate a spec file named config.json ``` We can edit the `config.json` to add customized behaviors for container. Here, we modify the `process` field to run `sleep 30`. ```json "process": { ... "args": [ "sleep", "30" ], ... } ``` Then we can explore the lifecycle of a container: ```bash cd .. # go back to the repository root sudo ./youki create -b tutorial tutorial_container # create a container with name `tutorial_container` sudo ./youki state tutorial_container # you can see the state the container is `created` sudo ./youki start tutorial_container # start the container sudo ./youki list # will show the list of containers, the container is `running` sudo ./youki delete tutorial_container # delete the container ``` Change the command to be executed in `config.json` and try something other than `sleep 30`. ### Rootless container `youki` provides the ability to run containers as non-root user([rootless mode](https://docs.docker.com/engine/security/rootless/)). To run a container in rootless mode, we need to add some extra options in `config.json`, other steps are same with above: ```bash mkdir -p tutorial/rootfs cd tutorial # use docker to export busybox into the rootfs directory docker export $(docker create busybox) | tar -C rootfs -xvf - ../youki spec --rootless # will generate a spec file named config.json with rootless mode ## Modify the `args` field as you like ../youki run rootless-container # will create and run a container with rootless mode ``` ## Usage Start the docker daemon. ```bash dockerd --experimental --add-runtime="youki=$(pwd)/youki" ``` If you get an error like the below, that means your normal Docker daemon is running, and it needs to be stopped. Do that with your init system (i.e., with systemd, run `sudo systemctl stop docker`, as root if necessary). ```console failed to start daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid ``` Now repeat the command, which should start the docker daemon. You can use youki in a different terminal to start the container. ```bash docker run -it --rm --runtime youki busybox ``` Afterwards, you can close the docker daemon process in other the other terminal. To restart normal docker daemon (if you had stopped it before), run: ```bash sudo systemctl start docker ``` ### Integration Tests Go and node-tap are required to run integration tests. See the [opencontainers/runtime-tools](https://github.com/opencontainers/runtime-tools) README for details. ```bash git submodule update --init --recursive just test-oci ``` ### Setting up Vagrant You can try youki on platforms other than Linux by using the Vagrantfile we have prepared. We have prepared two environments for vagrant, namely rootless mode and rootful mode ```bash git clone git@github.com:youki-dev/youki.git cd youki # If you want to develop in rootless mode, and this is the default mode vagrant up default vagrant ssh default # or if you want to develop in rootful mode vagrant up rootful vagrant ssh rootful # in virtual machine cd youki just youki-dev # or youki-release ``` # ๐Ÿ‘ฅ Community and Contributing Please refer to [our community page](https://youki-dev.github.io/youki/community/introduction.html). Thanks to all the people who already contributed! - - - **We are a [Cloud Native Computing Foundation](https://cncf.io/) sandbox project.** The Linux Foundationยฎ (TLF) has registered trademarks and uses trademarks. For a list of TLF trademarks, see [Trademark Usage](https://www.linuxfoundation.org/legal/trademark-usage). Copyright Contributors to youki, established as youki a Series of LF Projects, LLC. ## License [](https://app.fossa.com/projects/git%2Bgithub.com%2Fyouki-dev%2Fyouki?ref=badge_large) --- ## File: crates/libcontainer/README.md # libcontainer ### Building with musl In order to build with musl you must first remove the libseccomp dependency as it will reference shared libraries (`libseccomp`) which cannot be built with musl. Do this by using adding flags to Cargo. Use the `--no-default-features` flag followed by `-F` and whatever features you intend to build with such as `v2` as defined in Cargo.toml under features section. Next you will also need the `+nightly` flags when building with `rustup` and `cargo`. ```bash # Add rustup +nightly musl to toolchain rustup +nightly target add $(uname -m)-unknown-linux-musl # Build rustup +nightly stdlib with musl rustup +nightly toolchain install nightly-$(uname -m)-unknown-linux-musl # Build musl standard library cargo +nightly build -Zbuild-std --target $(uname -m)-unknown-linux-musl --no-default-features -F v2 cargo +nightly build --target $(uname -m)-unknown-linux-musl --no-default-features -F v2 ``` --- ## File: crates/liboci-cli/README.md # liboci-cli This is a crate to parse command line arguments for OCI container runtimes as specified in the [OCI Runtime Command Line Interface](https://github.com/opencontainers/runtime-tools/blob/master/docs/command-line-interface.md). ## Implemented subcommands | Command | liboci-cli | CLI Specification | runc | crun | youki | | :--------: | :--------: | :---------------: | :--: | :--: | :---: | | create | โœ… | โœ… | โœ… | โœ… | โœ… | | start | โœ… | โœ… | โœ… | โœ… | โœ… | | state | โœ… | โœ… | โœ… | โœ… | โœ… | | kill | โœ… | โœ… | โœ… | โœ… | โœ… | | delete | โœ… | โœ… | โœ… | โœ… | โœ… | | checkpoint | | | โœ… | โœ… | | | events | โœ… | | โœ… | | โœ… | | exec | โœ… | | โœ… | โœ… | โœ… | | features | โœ… | | โœ… | | | | list | โœ… | | โœ… | โœ… | โœ… | | pause | โœ… | | โœ… | โœ… | โœ… | | ps | โœ… | | โœ… | โœ… | โœ… | | restore | | | โœ… | โœ… | | | resume | โœ… | | โœ… | โœ… | โœ… | | run | โœ… | | โœ… | โœ… | โœ… | | spec | โœ… | | โœ… | โœ… | โœ… | | update | | | โœ… | โœ… | | --- ## File: docs/src/developer/e2e/containerd_integration_test_using_youki.md # containerd integration test using youki ## local ```console just containerd-test ``` --- ## File: docs/src/developer/e2e/e2e_tests.md # e2e tests There are various e2e tests: - [rust oci tests](./rust_oci_test.md) This is youki's original integration to verify the behavior of the low-level container runtime. - [containerd integration test](./containerd_integration_test_using_youki.md) This is the method that containerd's integration test runs with youki. - [runtime tools](./runtime_tools.md) This is the method to run the runtime tools that OCI manages as a test tool to verify meeting the OCI Runtime Spec - [Kubernetes test](./kubernetes_test.md) This test verifies youki works as a Kubernetes container runtime using Kind. --- ## File: docs/src/developer/e2e/integration_test.md # integration_test **Note** that these tests resides in `/tests/integration_test/` at the time of writing. This crate contains the Rust port of OCI-runtime tools integration tests, which are used to test if the runtime works as per the OCI spec or not. Initially youki used the original implementation of these test provided in the OCI repository [here](https://github.com/opencontainers/runtime-tools/tree/master/validation). But those tests are written in Go, which made the developers depend on two language environments Rust and Go to compile youki and test it. The Validation tests themselves also have an optional dependency on node js to parse their output, which can make it a third language dependency. Other than that, those tests also showed some issues while running on some local systems, and thus running the tests would be difficult on local system. As the runtime is a complex piece of software, it becomes useful to have a set of tests that can be run with changes in code, so one can verify that change in one part of youki has not accidentally broken some other part of youki. Thus we decided to port the tests to Rust, and validate them, so that we have a set of unit tests as well of integration tests to validate the working of runtime. These tests are still under development, and you can check the [tracking issue](https://github.com/youki-dev/youki/issues/361) for more details. More details on working of these tests can be found at [https://github.com/youki-dev/youki/tree/main/crates/integration_test](https://github.com/youki-dev/youki/tree/main/tests/contest/contest). As these tests are under development, these are validated on a standard runtime such as runc in the GitHub CI, so validate the tests themselves. ## Notes ### About the create container function The test_utils provides a `create_container` function which can be used to run the `youki create` command. It returns the child process struct, which can be either `wait()` or `wait_with_output()` to wait for it finishing. Unless you know what are you doing, it is recommended to call `wait()` on it, as otherwise the process will hang. As explained in the [youki docs](../youki.md) , the `youki create` process, after starting forks, and the forked process keeps waiting for another youki process to send it the `start` signal , and after receiving it, that forked process execs the container program. If you are simply trying to create a container, such as in case of `test_outside_runtime` then calling `wait_with_output()` will cause it to hang. If you are actually going to start a container, and need output from the container process, then you must keep the `Child struct` returned by `create` function and call `wait_with_output()` on it **AFTER** you have called the start command on that container, which will give you the `stdout` and `stderr` of the process running inside the container. To understand how this works, take a look at [handling stdio](https://github.com/opencontainers/runc/blob/master/docs/terminals.md) of the runc, specially the [detached pass-through mode](https://github.com/opencontainers/runc/blob/master/docs/terminals.md#detached-pass-through) section. As explained in it, we setup the stdio for the original youki process in `youki create` by setting the stdio to `Stdio::piped()` in the `create` function. Then we set the `terminal` option to `false` (which is the default anyway) in the spec, which makes it run in the pass-through mode. Then when the create process is done its work, and its forked process is waiting for the start signal, it uses the same stdio pipes. Thus calling `wait_with_output()` without starting will keep it hanged up, and after calling start, stdio of the program to be run inside the container can be obtained from the `youki create`'s process. ### How test inside container works We use `test_inside_container` for making sure that the restrictions and constraints are uphold from inside the container process. For that, first whichever integration test needs to use it, must define the runtimetest as the container process in the spec, and then use `test_inside_container` function. It requires a function which will do the necessary setup for the tests that are to be run inside. Then the counterpart for the test should be added to the `runtimetest` crate, which will run inside the container and if there is any error, print it to the `stderr`. The `test_inside_container` function will wait for the tests to be over and then check the `stderr` to be empty. If it is not, the test is assumed to fail. --- ## File: docs/src/developer/e2e/runtime_tools.md # runtime tools ## local ```console $ git submodule update --init --recursive $ just test-oci ``` --- ## File: docs/src/developer/e2e/runtimetest.md # Runtime Test **Note** that these tests resides in /tests/runtimetest at the time of writing. This crate provides a binary which is used by integration tests to verify that the restrictions and constraints applied to the container are upheld by the container process, from inside the container process. This runs the tests one-by-one, and the failing test prints the error to the stderr. ## Notes This binary must be compiled with the option of static linking to crt0 given to the rustc. If compiled without it, it will add a linking to /lib64/ld-linux-x86-64.so . The binary compiled this way cannot be run inside the container process, as they do not have access to /lib64/... Thus the runtime test must be statically linked to crt0. While developing, originally this was added to the common workspace of all crates in youki. But then it was realized that this was quite inefficient because : - All packages except runtimetest will be compiled with dynamic linking - Runtimetest will be compiled with static linking Now runtimetest needs at least `oci-spec` and `nix` package for its operations, which are also dependencies of other packages in the workspace. Thus both of these, and recursively their dependencies must be compiled twice, each time, once for dynamic linking and once for static. The took a long time in the compilation stage, especially when developing / adding new tests. Separating runtimetest from the workspace allows it to have a separate target/ directory, where it can store the statically compiled dependencies, and the workspace can have its target/ directory, where it can store its dynamically compiled dependencies. That way only the crates which have changes need to be compiled (runtimetest or integration test), and not their dependencies. In case in future this separation is not required, or some other configuration is chosen, make sure the multiple compilation issue does not arise, or the advantages of new method outweigh the time spent in double compilation. To see if a binary can be run inside the container process, run ```console readelf -l path/to/binary |grep "program interpreter" ``` `[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]` means that the binary is not statically linked, and cannot be run inside the container process. If the above command gives no output, that means it does not require any program interpreter and can be run inside the container. Another way is to run ```console file path/to/binary ``` ```console ./youki: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=...., for GNU/Linux 3.2.0, with debug_info, not stripped` ``` This output indicates that the binary is dynamically linked, thus cannot be run inside the container process ```console ./runtimetest: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=...., for GNU/Linux 3.2.0, with debug_info, not stripped ``` This output indicates that the binary is statically linked, and can be run inside the container process Some links to help : - [how to generate static executable](https://stackoverflow.com/questions/31770604/how-to-generate-statically-linked-executables) - [understanding the error which dynamically linked library gives](https://superuser.com/questions/248512/why-do-i-get-command-not-found-when-the-binary-file-exists) - [Rust cargo config for rustflags](https://doc.rust-lang.org/cargo/reference/config.html) --- ## File: docs/src/developer/e2e/rust_oci_test.md # Contest This is youki's original integration to verify the behavior of the low-level container runtime. # How to run ```console just test-contest ``` # How to write We will not go into detail here, but will explain how to write and add a new test case based on an example test. Fully the code of the example test ```rust,no_run,noplayground {{#include ../../../../tests/contest/contest/src/tests/example/hello_world.rs}} ``` 1. Build the OCI Runtime Spec you want to verify This testing framework automatically places [runtimetest](./runtimetest.md) in the container. In other words, you can test the processes you want to execute within a container by writing them in runtimetest. Therefore, it is common practice here to write an OCI Runtime Spec that executes `runtimetest`. ```rust,no_run,noplayground {{#include ../../../../tests/contest/contest/src/tests/example/hello_world.rs:get_example_spec}} ``` 2. Prepare a function that returns a `TestResult`, which represents the result of the test. ```rust,no_run,noplayground {{#include ../../../../tests/contest/contest/src/tests/example/hello_world.rs:example_test}} ``` 3. Create a `TestGroup` and register a test case you created ```rust,no_run,noplayground {{#include ../../../../tests/contest/contest/src/tests/example/hello_world.rs:get_example_test}} ``` 4. Register the `TestGroup` you created to a `TestManager` ```rust,no_run,noplayground {{#include ../../../../tests/contest/contest/src/main.rs:register_example_test}} ``` 5. Write the validation you want to run within a test container ```rust,no_run,noplayground {{#include ../../../../tests/contest/runtimetest/src/main.rs:example_runtimetest_main}} ``` ```rust,no_run,noplayground {{#include ../../../../tests/contest/runtimetest/src/tests.rs:example_hello_world}} ```