CubeSandbox Documentation

Instant, Concurrent, Secure & Lightweight Sandbox for AI Agents.

11,016 stars Rust Rust 1 file · ~39,052 tokens #agents#container#sandbox
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy CubeSandbox on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run)
docker run -d --name cubesandbox --restart always tencentcloud/cubesandbox:latest

File: README.md

CubeSandbox


Cube Sandbox is a high-performance, out-of-the-box secure sandbox service built on RustVMM and KVM. It supports both single-node deployment and easy scaling to multi-node clusters. It is compatible with the E2B SDK and can create a hardware-isolated, fully serviceable sandbox in under 60ms with less than 5MB of memory overhead.

📰 News

| | v0.6: K8s deploy, Volume framework, template aliases
K8s deploy — Deploy Cube control-plane components and compute nodes on Kubernetes
Volume framework — E2B-compatible Volume framework that lets users plug in custom backend storage
Template aliases — Set an alias when creating a template, and create sandboxes by specifying that alias.
Changelog → ·
K8s deploy → ·
Volume plugin → |

| --- | --- |
| | v0.5: AutoPause, Terraform deployer, ARM64 & network policy hardening
AutoPause/AutoResume — idle sandboxes auto-suspend and wake on the next request. Terraform one-click cluster deploy ARM64 native full-stack support network policy hardening — per-sandbox traffic tokens, policy-routing egress.
Changelog → ·
Terraform deploy → |
| | v0.4: Safer egress, easier ops
Credential vault — Agents call external APIs as usual; keys never enter the sandbox. Dashboard — version matrix and template health checks; see at a glance whether templates need rebuilding after upgrades.
Changelog → ·
Security proxy guide → ·
WebUI guide → |
| | Snapshot, Clone & Rollback at hundred-millisecond granularity
CubeSandbox 0.3.0 introduces the CubeCoW Copy-on-Write snapshot engine, enabling event-level snapshots, instant cloning, and rollback to any saved state.
Changelog → |
| | 🎉 Initial open-source release
Cube Sandbox is now open source! Millisecond boot, hardware-level isolation, E2B-compatible sandbox for AI Agents.
Changelog → |

Product Highlights

| ⚡ Ultra-fast Startup
Resource pooling and snapshot cloning skip all cold-start overhead. Average <60ms cold start — sandbox creation faster than a blink.
Quick start → | 🔒 Hardware Isolation
Every sandbox runs a dedicated OS kernel in its own MicroVM.
Architecture → | 🔌 E2B SDK Compatible
Compatible with E2B SDK interface. Switch from E2B Cloud seamlessly by changing one environment variable — zero client code changes.
Examples → |

| --- | --- | --- |
| 📦 High-density Deployment
<5MB overhead per sandbox enables thousands of instances per server via kernel sharing and Copy-on-Write (CoW). Supports automatic sandbox pause and resume, further improving deployment density and cost optimization.
Quick start → | 🛡️ Network Security
eBPF-based inter-sandbox isolation and egress filtering at kernel level; built-in L7 security proxy enables per-domain/path/method policies with automatic credential injection — secrets never visible to sandbox code.
Security proxy guide → | 📸 Flexible State Management
High-frequency snapshot and rollback at hundred-millisecond granularity. Create checkpoints on running sandboxes, roll back to any saved state at any time, or fork from a specific state to explore in parallel.
v0.3 changelog → |
| 💾 Volume Framework
E2B-compatible Volume framework that lets users plug in custom backend storage solutions. Volumes have an independent lifecycle and can be shared across sandboxes.
Volume plugin → | 🚀 Production Deployment
Deploy production clusters on Tencent Cloud with one click using Terraform. Also supports deployment on standard Kubernetes clusters (preview).
Terraform deploy → ·
K8s deploy → | 💪 ARM Architecture Support
Full native ARM64 support across compilation, build, and deployment workflows.
Bare-metal deploy → |

Demos

Installation & Demo Performance Test RL (SWE-Bench) Snapshot · Clone · Rollback

Benchmarks

In the context of AI Agent code execution, CubeSandbox achieves the perfect balance of security and performance:

Metric Docker Container Traditional VM CubeSandbox
Isolation Level Low (Shared Kernel Namespaces) High (Dedicated Kernel) Extreme (Dedicated Kernel + eBPF)
Boot Speed *Full-OS boot duration 200ms Seconds Sub-millisecond (<60ms)
Memory Overhead Low (Shared Kernel) High (Full OS) Ultra-low (Aggressively stripped, <5MB)
Deployment Density High Low Extreme (Thousands per node)
E2B SDK Compatible / / ✅ Drop-in
  • Cold start benchmarked on bare-metal. 60ms at single concurrency; under 50 concurrent creations, avg 67ms, P95 90ms, P99 137ms — consistently sub-150ms.
  • Memory overhead measured with sandbox specs ≤ 32GB. Larger configurations may see a marginal increase.

For detailed metrics on startup latency and resource overhead, see the Core Operations Performance Benchmark Report (bare metal) and the PVM Cloud Server Benchmark Report.

Sub-150ms sandbox delivery under both single and high-concurrency workloads CubeSandbox base memory footprint across various instance sizes
text
(*Blue: Sandbox specifications; Orange: Base memory overhead). Note that memory consumption increases only marginally as instance sizes scale up. |  |

Quick Start

Cube Sandbox requires an x86_64 Linux environment with KVM support.

The guide walks you through everything in four steps — provisioning a server, installing Cube Sandbox, creating a sandbox template, and running your first agent code. No source build needed, up and running in minutes.

| 🖥 PVM · Cloud VM →

text
**🏆 Recommended** | 🏗 Bare Metal → | 💻 Dev-Env →
  
  
  ⚠️ **Not recommended — poor performance** |

| --- | --- | --- |

First thing after install: open the Web console

After one-click deployment, open in your browser:

text
http://<control-node IP>:12088

Recommended three steps:

  1. Check overview — Open Overview, confirm nodes are Ready and capacity looks healthy
  2. Prepare a template — Install an official preset from Template Store; skip if you already have a READY template under Templates
  3. Create a sandbox — Sandboxes → + New sandbox, pick a READY template, and view live logs on the detail page within seconds

See the full WebUI console guide.

Deep Dive

Architecture

Component Responsibility
CubeAPI High-concurrency REST API Gateway (Rust), compatible with E2B. Swap the URL for seamless migration.
CubeMaster Cluster orchestrator. Receives API requests and dispatches them to corresponding Cubelets. Manages resource scheduling and cluster state.
CubeProxy Reverse proxy, compatible with the E2B protocol, routing requests to the appropriate sandbox instances.
Cubelet Compute node local scheduling component. Manages the complete lifecycle of all sandbox instances on the node.
CubeVS eBPF-based virtual switch, providing kernel-level network isolation and security policy enforcement.
CubeEgress OpenResty-based egress security gateway: L7 domain filtering, credential injection, and access auditing; works with CubeVS kernel policies so sandbox traffic cannot bypass inspection.
CubeHypervisor & CubeShim Virtualization layer — CubeHypervisor manages KVM MicroVMs, CubeShim implements the containerd Shim v2 API to integrate sandboxes into the container runtime.

👉 For more details, please read the Architecture Design Document and CubeVS Network Model.

Community & Contributing

We welcome contributions of all kinds—whether it's a bug report, feature suggestion, documentation improvement, or code submission!

  • 🐞 Found a Bug or have questions? Submit an issue on GitHub Issues.
  • 💡 Have an Idea? Join the conversation in GitHub Discussions.
  • 🛠️ Want to Code? Check out our CONTRIBUTING.md to learn how to submit a Pull Request.
  • 📝 Want to contribute docs? Submit bilingual PRs to our community doc channels: Troubleshooting, Use Cases, and Integrations. Additionally, the Cube 100 Program is now open — we're looking for the first 100 teams running AI agents in production with Cube. Limited to 100 seats. Learn more & apply →
  • 💬 Want to Chat? Join our Discord.

Roadmap

Coming soon — see the full roadmap for details.

Feature Description
Cross-Node Pause & Resume Suspend a sandbox on one node and resume it on another with full memory and filesystem state preserved
Cross-Node Snapshot-Based Sandbox Launch Create new sandboxes from a snapshot on a different node than the one where it was taken, backed by cluster-shared, on-demand-loading storage
E2B API Compatibility Close remaining gaps with the E2B specification for full drop-in compatibility
Control Plane / Data Plane Separation Decouple the control plane from the data plane so control plane upgrades or failures never affect sandboxes already in flight
Sandbox Fault Recovery Automatic detection and recovery of crashed VMs, stuck shim processes, and network partitions with configurable recovery policies
Scheduling & Operations Enhancements Resource-aware placement, affinity rules, live rebalancing, and node drain with sandbox migration

Sponsors

Contributors

Thanks to all contributors:

License

CubeSandbox is released under the Apache License 2.0.

The birth of CubeSandbox stands on the shoulders of open-source giants. Special thanks to Cloud Hypervisor, Kata Containers, virtiofsd, containerd-shim-rs, ttrpc-rust, and others. We have made tailored modifications to some components to fit the CubeSandbox execution model, and the original in-file copyright notices are preserved.



File: agent/libs/safe-path/README.md

Safe Path

A library to safely handle filesystem paths, typically for container runtimes.

There are often path related attacks, such as symlink based attacks, TOCTTOU attacks. The safe-path crate
provides several functions and utility structures to protect against path resolution related attacks.

Support

Operating Systems:

  • Linux

Reference

License

This code is licensed under Apache-2.0.


File: agent/libs/README.md

The src/libs directory hosts library crates which may be shared by multiple Kata Containers components
or published to crates.io.

Library Crates

Currently it provides following library crates:

| Library | Description |
|-|-|-|
| logging | Facilities to setup logging subsystem based slog. |
| safe-path | Utilities to safely resolve filesystem paths. |


File: agent/README.md

cube-agent

The in-VM guest agent for Cube Sandbox. It runs as PID 1 inside each MicroVM
(after cube-init execs it with wrapper_mode=on) and manages the full
container lifecycle within the sandbox.

Upstream

cube-agent is derived from the kata-containers/kata-containers agent (src/agent/), originally authored by the Kata Containers community and licensed under Apache-2.0.

Cube-specific modifications include:

  • Replaced the Kata runtime protocol with the Cube ttrpc API (libs/protocols/)
  • Added cube/ sub-crate for Cube-specific sandbox extensions (custom device management, configuration handling)
  • Adapted vsock port assignments and boot flow to match the Cube hypervisor and shim

The original Kata Containers copyright notices and license headers are preserved in all modified source files.

Role in the System

text
Host
┌────────────────────────────────┐
│  containerd-shim-cube-rs       │
│  pmem0 = guest OS (cube-init)  │
│  pmem1 = cube-agent.ext4       │
│       │  ttrpc over vsock      │
└───────┼────────────────────────┘
        │  (vsock channel)
   MicroVM boundary
        │
┌───────▼────────────────────────┐
│  cube-init (/sbin/init)        │
│    mounts /dev/pmem1 →         │
│      /run/support (ext4,ro,dax)│
│    exec /run/support/cube-agent│
│       │                        │
│  cube-agent (PID 1, wrapper)   │
│  ┌────▼──────────────────────┐ │
│  │  container workload       │ │
│  └───────────────────────────┘ │
└────────────────────────────────┘

cube-agent is packaged as an independent cube-agent.ext4 plane file
(containing only /cube-agent). CubeShim injects it as virtio-pmem1. Guest
Image contains lightweight cube-init as /sbin/init, which mounts pmem1 and
execs the agent with wrapper_mode=on so the agent skips duplicate
general_mount.

When the MicroVM boots, the agent:

  1. Skips init mounts when wrapper_mode=on — cube-init already prepared /proc, /sys, /run, etc.
  2. Notifies shim via SysCtrl — writes VsockServerReady (x86 PIO 0x680 / aarch64 MMIO 0x0903_0000). This is not the template SysStart handshake; open-source cube-init does not perform snapshot-mode SysCtrl handshake.
  3. Listens for ttrpc commands — exposes the Cube agent API over a vsock channel; the shim (containerd-shim-cube-rs) connects to this channel to drive the agent.
  4. Manages container lifecycle — handles CreateContainer / StartContainer / ExecProcess / SignalProcess / RemoveContainer requests, delegating to rustjail for OCI-compliant container execution.
  5. Forwards I/O — proxies container stdio streams back to the shim over vsock.
  6. Exposes metrics — exports Prometheus-compatible metrics for guest CPU, memory, and container health.

Repository Layout

text
agent/
├── src/             # Agent binary (main entry point, ttrpc server, sandbox/container state)
│   ├── main.rs      # Startup, vsock listener, ttrpc server init
│   ├── rpc.rs       # ttrpc service handlers (implements the Cube agent API)
│   ├── sandbox.rs   # Sandbox state management
│   ├── mount.rs     # Filesystem mount handling
│   ├── network.rs   # Guest network configuration
│   └── ...
├── rustjail/        # OCI container runtime primitives (namespaces, cgroups, seccomp)
├── cube/            # Cube-specific extensions (device model, config)
├── libs/
│   ├── protocols/   # ttrpc API definitions (protobuf) and generated Rust bindings
│   ├── oci/         # OCI spec types
│   ├── logging/     # slog-based logging setup
│   └── safe-path/   # Safe filesystem path utilities
├── vsock-exporter/  # Prometheus metrics exporter over vsock
├── bootstrap.sh     # One-time musl toolchain setup
└── build.sh         # Release build script (musl static binary)

See also ../guest-init/ for the lightweight Guest PID 1 (cube-init).

Build

cube-agent is built as a statically linked musl binary, then packaged into
cube-agent.ext4 by deploy/one-click/build-agent-ext4.sh.

Prerequisites

bash
# Install the musl target and bootstrap system dependencies (run once)
arch=$(uname -m)
rustup target add "${arch}-unknown-linux-musl"
sudo ln -s /usr/bin/g++ /bin/musl-g++
sudo bash bootstrap.sh

Compile

bash
bash build.sh

The output binary is placed at target/<arch>-unknown-linux-musl/release/cube-agent.

bash
make all-docker
# or from repo root:
make agent
bash
# From repo root — builds musl-static cube-agent inside the builder image,
# then packages _output/cube-agent/{cube-agent.ext4,version}:
make agent-ext4
# aliases:
make cube-agent-ext4
# with guest PID1 as well:
make pmem-assets

Or call the script directly after a prebuilt binary is available:

bash
OUTPUT_DIR=/tmp/cube-agent ./deploy/one-click/build-agent-ext4.sh

API

cube-agent exposes a ttrpc API over vsock. The protocol is defined in protobuf files under libs/protocols/protos/ and is shared with the Cube runtime (CubeShim).

To regenerate protocol bindings after changing .proto files:

bash
# Rust bindings (auto-generated at build time by build.rs)
cargo build

# Go bindings (used by the Cube runtime side)
make generate-protocols   # requires protoc in $PATH

To install protoc:

bash
# Debian/Ubuntu
sudo apt-get install -y protobuf-compiler

# Fedora/CentOS/RHEL
sudo dnf install -y protobuf-compiler

License

Apache-2.0 — see LICENSE for details.

This component incorporates code from the Kata Containers project, © The Kata Containers Authors, licensed under Apache-2.0.


File: cube-lifecycle-manager/README.md

cube-lifecycle-manager

Standalone service that owns sandbox auto-pause / auto-resume coordination
between CubeMaster, CubeProxy and Redis.

  • Consumes lifecycle events from cube:v1:shared:sandbox:lifecycle:events
  • Discovers every live CubeProxy replica in real time via
    cube:v1:shared:cube_proxy:{registry,heartbeat} and broadcasts sandbox
    metadata + state through their /admin/* endpoints
  • Handles the synchronous /internal/resume callback CubeProxy invokes when
    a paused sandbox receives a request

Local development

sh
make test    # go test ./...
make build   # local image, tag: cube-lifecycle-manager:v0.6.0-<arch>

Publishing images

Multi-arch release flow (mirrors CubeEgress/Makefile). Runs against both
cube-sandbox-int.tencentcloudcr.com and cube-sandbox-cn.tencentcloudcr.com.

sh
# On an amd64 host:
make build push ARCH=amd64

# On an arm64 host:
make build push ARCH=arm64

# On either host (both arch-specific images must already be pushed):
make manifest

Overrides:

  • IMAGE_TAG=<tag> — override the release tag (default v0.6.0)
  • V=1 — verbose docker commands

Configuration

All configuration is via environment variables (prefix CUBE_LCM_); see
internal/config/config.go for the authoritative list.


File: CubeAPI/examples/readme.md

Examples

See the top-level examples/ directory for standalone, runnable examples:


File: cubecow/README.md

cubecow

A Copy-On-Write storage engine built on xfs-reflink (FICLONE), shipped as a Rust library crate that provides thin-provisioned volume management and O(1) snapshot/clone capabilities.

Overview

cubecow is a library crate that exposes a uniform interface through the Engine trait. It currently ships with a single backend:

  • xfs-reflink (ReflinkEngine) — uses regular files as volumes and snapshots inside a directory on a FICLONE-capable filesystem (XFS with reflink=1 / Btrfs / OCFS2 / …), and relies on the kernel FICLONE ioctl for O(1) metadata snapshots.

Key Features

Feature Description
O(1) snapshots Powered by the kernel FICLONE ioctl: extents are shared rather than byte-copied
Flat snapshot model Every snapshot is an independent file; deleting one snapshot never affects any other
O(1) clones Taking a snapshot of a snapshot is also a FICLONE — cost depends only on the source file's extent count
Simple crash recovery Volumes and snapshots are files; on restart the in-memory index is rebuilt by scanning <root_dir>/volumes/. No standalone ledger required
Backend-agnostic API All callers go through dyn Engine; adding a new backend does not require any change to the FFI / SDK callers

Architecture

text
┌──────────────────────────────────────────────────────────┐
│                Public API layer (lib.rs)                  │
│   Engine trait + Volume / Snapshot / VolumeBlockInfo /   │
│           initialize / initialize_without_logging       │
├──────────────────────────────────────────────────────────┤
│                C FFI layer (ffi.rs)                       │
│   #[no_mangle] extern "C" fn cubecow_*(...)              │
│   → header file include/cubecow.h                         │
├──────────────────────────────────────────────────────────┤
│              Backend implementation (engine/)             │
│   engine::reflink::ReflinkEngine                          │
│       ├── name_index: RwLock<HashMap<String, VolumeInfo>> │
│       ├── volumes_dir: <root_dir>/volumes/                │
│       └── direct FICLONE / unlink / fsync_dir calls       │
├──────────────────────────────────────────────────────────┤
│              Infrastructure (pkg/)                        │
│   errors / logger / metrics                               │
└──────────────────────────────────────────────────────────┘

On-disk layout for volumes and snapshots

text
<root_dir>/                         # AppConfig.backend.reflink.root_dir
└── volumes/
    ├── <vol-A>/
    │   ├── <vol-A>          # main volume file, FICLONE source
    │   ├── <snap-A1>        # snapshot derived from vol-A
    │   └── <snap-A2>        # snapshot derived from vol-A
    └── <vol-B>/
        ├── <vol-B>
        └── <snap-B1>

Every snapshot is "flattened" into the directory of its ultimate origin volume; deleting a snapshot is a single unlink and never touches any other snapshot's extents.

Build

Prerequisites

  • Rust 1.93+ (installation via rustup is recommended)
  • A FICLONE-capable filesystem to host root_dir (production setups should use mkfs.xfs -m reflink=1,crc=1)

Compile

bash
# Debug
cargo build

# Release (produces target/release/libcubecow.{so,a})
cargo build --release

Unit tests / benchmarks

bash
# Unit tests (no root required; nothing is actually mkfs'd or mounted)
cargo test --lib

# End-to-end reflink performance benchmark (requires root + xfsprogs + losetup)
sudo cargo bench --bench reflink_ops

Configuration

Both TOML and JSON are supported; the JSON form is consumed directly by the FFI entry points cubecow_init_*_from_json as a string. The schema is intentionally minimal:

TOML

toml
[log]
level = "info,h2=warn,hyper=warn"
format = "compact"               # "json" | "compact" | "pretty"
file = "/var/log/cubecow.log"    # optional; logs go to stdout when unset
rotation = "daily"               # "daily" | "hourly" | "never"

[backend]
kind = "reflink"                 # the only backend currently supported

[backend.reflink]
root_dir = "/var/lib/cubecow/reflink"

JSON

json
{
  "log": {
    "level": "info",
    "format": "compact"
  },
  "backend": {
    "kind": "reflink",
    "reflink": { "root_dir": "/var/lib/cubecow/reflink" }
  }
}

root_dir must be an absolute path whose mount point supports FICLONE. During initialize the engine performs a one-shot FICLONE probe; if it fails, startup is rejected outright.

Rust SDK usage

rust
use cubecow::{config::AppConfig, Engine};

let config = AppConfig::load("/etc/cubecow.toml")?;
let engine: Box<dyn Engine> = cubecow::initialize(config)?;

// Create a volume
let vol = engine.create_volume("my-vol", 10 * 1024 * 1024 * 1024)?;
println!("device path: {}", vol.device_path);

// Create a snapshot (metadata-only; do not activate immediately)
let snap = engine.create_snapshot("my-vol", "my-snap", false)?;

// List snapshots
let (snaps, _next) = engine.list_snapshots("my-vol", 10, None);

C FFI / Go integration

The C header lives at include/cubecow.h. The cubecow_ function prefix is preserved to keep ABI compatibility. A typical call flow:

c
#include "cubecow.h"

CubecowEngineHandle eng = cubecow_init_from_json(json_blob);
if (!eng) {
    fprintf(stderr, "init failed: %s\n", cubecow_last_error());
    return 1;
}

char* dev = NULL;
int rc = cubecow_create_volume(eng, "my-vol", 10ULL << 30, &dev);
if (rc == COW_OK) {
    printf("created at %s\n", dev);
    cubecow_free_string(dev);
}
cubecow_shutdown(eng);

A Go example lives at examples/go-test/ and uses cgo to link against -lcubecow:

bash
# 1. Build the Rust shared lib
cargo build --release

# 2. Build & run the Go test binary
cd examples/go-test
make run

File: Cubelet/doc/cubelet-api.md

Protocol Documentation

Table of Contents


Top

api/services/cubebox/v1/cubebox.proto

AppSnapshotRequest

AppSnapshotRequest is the request for creating an app snapshot.
This request creates a cubebox, makes an app snapshot using cube-runtime,
and then destroys the cubebox.

Field Type Label Description
create_request RunCubeSandboxRequest The RunCubeSandboxRequest to create the cubebox. Required annotations: - cube.master.appsnapshot.create: "true" - cube.master.appsnapshot.template.id: "<template_id>"
snapshot_dir string Custom snapshot directory path. If empty, uses default path.

AppSnapshotResponse

AppSnapshotResponse is the response for app snapshot creation.

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxID string The sandbox ID that was created and destroyed.
templateID string The template ID from the request annotations.
snapshotPath string The snapshot path where the snapshot was created.
rootfs_vol string Authoritative cubecow rootfs object name.
memory_vol string Authoritative cubecow memory object name.
rootfs_kind string Authoritative cubecow rootfs object kind.
memory_kind string Authoritative cubecow memory object kind.
rootfs_size_bytes uint64 Rootfs size at snapshot time, used by repair and validation.
guest_image_version string guest-image version bound when this snapshot was created.
agent_version string cube-agent version bound when this snapshot was created.
kernel_version string Guest kernel artifact identity bound when this snapshot was created.
envd_version string envd semantic version collected in-guest at snapshot time (best-effort).
shim_version string cube-shim (and sibling cube-runtime) version bound when this snapshot was created.

CDIDevice

CDIDevice specifies a CDI device information.

Field Type Label Description
name string Fully qualified CDI device name for example: vendor.com/gpu=gpudevice1 see more details in the CDI specification: https://github.com/container-orchestrated-devices/container-device-interface/blob/main/SPEC.md

Capability

Capability contains the container capabilities to add or drop

Field Type Label Description
add_capabilities string repeated List of capabilities to add.
drop_capabilities string repeated List of capabilities to drop.
add_ambient_capabilities string repeated List of ambient capabilities to add.

CleanupOrphanStorageFilesRequest

Field Type Label Description
requestID string requestID reqID
bucket string Optional bucket name; defaults to "emptydir/v1" when empty.
formats string repeated Format roots cubelet should scan, e.g. "512Mi" / "1Gi" / "others". When empty cubelet falls back to a built-in default list.
dry_run bool When true cubelet only reports orphans without removing them.

CleanupOrphanStorageFilesResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
orphans StorageOrphanEntry repeated Orphan files cubelet found (and possibly removed).

CleanupTemplateRequest

Field Type Label Description
requestID string requestID reqID
templateID string Logical template ID.
snapshotPath string Snapshot path recorded on this node, if any. DEPRECATED in v4: cubelet resolves this from local catalog; legacy masters may still send it for backward compatibility but new masters MUST send empty.
objects CowObjectRef repeated cubecow objects that should be removed on this node. DEPRECATED in v4: cubelet derives objects from local catalog; legacy masters may still populate this for backward compatibility but new masters MUST send empty.

CleanupTemplateResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
templateID string Logical template ID.
plugin_volume_ids string repeated Deprecated: unused. Pause/Resume volume refcounts follow normal Create/Destroy ExtInfo events; CleanupTemplate does not adjust them.

CommitSandboxRequest

Field Type Label Description
requestID string requestID reqID
sandboxID string ID of the running sandbox to snapshot.
templateID string Logical template ID.
snapshot_dir string Custom snapshot directory path. If empty, uses default path.

CommitSandboxResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxID string ID of the sandbox that was snapshotted.
templateID string The logical template ID.
snapshotPath string The snapshot path where the snapshot was created.
rootfs_vol string Authoritative cubecow rootfs object name.
memory_vol string Authoritative cubecow memory object name.
rootfs_kind string Authoritative cubecow rootfs object kind.
memory_kind string Authoritative cubecow memory object kind.
rootfs_dev string Diagnostic-only rootfs device path; re-resolve from rootfs_vol on use.
memory_dev string Diagnostic-only memory device path; re-resolve from memory_vol on use.
rootfs_size_bytes uint64 Rootfs size at snapshot time, used by rollback resize planning.
guest_image_version string guest-image version bound when this snapshot was created.
agent_version string cube-agent version bound when this snapshot was created.
kernel_version string Guest kernel artifact identity bound when this snapshot was created.
envd_version string envd semantic version collected in-guest at commit time (best-effort).
shim_version string cube-shim (and sibling cube-runtime) version bound when this snapshot was created.

Container

Field Type Label Description
id string ID of the container, used by the container runtime to identify a container.
image string Spec of the image.
type string Type is container type. Sandbox or Container.
resources Resource Resources specification for the container.
state ContainerState State of the container.
created_at int64 Creation time of the container in nanoseconds.
finished_at int64 exit time of the container in nanoseconds.
labels Container.LabelsEntry repeated
paused_at int64
volume_mounts VolumeMounts repeated Mounts declared on the container spec (including host-dir mounts).

Container.LabelsEntry

Field Type Label Description
key string
value string

ContainerConfig

Field Type Label Description
name string Name of the container
image cubelet.services.images.v1.ImageSpec Image to use.
command string repeated Command to execute (i.e., entrypoint for docker)
args string repeated Args for the Command (i.e., command for docker)
working_dir string Current working directory of the command.
envs KeyValue repeated List of environment variable to set in the container.
volume_mounts VolumeMounts repeated Mounts for the container.
r_limit RLimit r_limit for container
resources Resource Resources specification for the container.
security_context ContainerSecurityContext security_context configuration for the container.
probe Probe probe
sysctls ContainerConfig.SysctlsEntry repeated sysctls holds linux sysctls config for the container.
syscalls SysCall repeated syscalls holds linux syscall config for the container.
dns_config DNSConfig DNS config for the container.
annotations ContainerConfig.AnnotationsEntry repeated annotations
hostAliases HostAlias repeated HostAliases is an optional list of hosts and IPs that will be injected into the sandbox's hosts file if specified.
prestop PreStop PreStop
poststop PostStop PostStop
id string container name use by cubebox
hooks Hooks Hooks for the container.
oci_config OCIConfig CDI

ContainerConfig.AnnotationsEntry

Field Type Label Description
key string
value string

ContainerConfig.SysctlsEntry

Field Type Label Description
key string
value string

ContainerSecurityContext

ContainerSecurityContext holds linux security configuration that will be applied to a container.

Field Type Label Description
capabilities Capability Capabilities to add or drop.
privileged bool If set, run container in privileged mode. Privileged mode is incompatible with the following options. If privileged is set, the following features MAY have no effect: 1. capabilities 2. selinux_options 4. seccomp 5. apparmor

Privileged mode implies the following specific options are applied: 1. All capabilities are added. 2. Sensitive paths, such as kernel module paths within sysfs, are not masked. 3. Any sysfs and procfs mounts are mounted RW. 4. AppArmor confinement is not applied. 5. Seccomp restrictions are not applied. 6. The device cgroup does not restrict access to any devices. 7. All devices from the host's /dev are available within the container. 8. SELinux restrictions are not applied (e.g. label=disabled). |
| run_as_user | Int64Value | | UID to run the container process as. Only one of run_as_user and run_as_username can be specified at a time. |
| no_new_privs | bool | | no_new_privs defines if the flag for no_new_privs should be set on the container. |
| run_as_group | Int64Value | | GID to run the container process as. run_as_group should only be specified when run_as_user or run_as_username is specified; otherwise, the runtime MUST error. |
| run_as_username | string | | User name to run the container process as. If specified, the user MUST exist in the container image (i.e. in the /etc/passwd inside the image), and be resolved there by the runtime; otherwise, the runtime MUST error. |
| readonly_rootfs | bool | | If set, the root filesystem of the container is read-only. when not set,RunCubeSandboxRequest.volumes should provide one empty_dir source for the rootfs rw and Container.VolumeMounts provide container_path="/" format |

ContainerStateValue

ContainerStateValue is the wrapper of ContainerState.

Field Type Label Description
state ContainerState State of the container.

CowObjectRef

Field Type Label Description
name string Authoritative cubecow object name.
kind string cubecow object kind: "snapshot" or "volume".
role string Logical role: "rootfs", "memory", "build_rootfs", etc.

CowObjectStatus

Field Type Label Description
name string Authoritative cubecow object name.
kind string cubecow object kind: "snapshot" or "volume".
role string Logical role.
exists bool Whether the object currently exists on this node.
device_path string Current device path if resolvable.
size_bytes uint64 Current size in bytes if resolvable.
error_message string Object-specific inspection error if any.

CubeNetworkConfig

Field Type Label Description
allow_internet_access bool optional
allow_out string repeated
deny_out string repeated
rules EgressRule repeated L7 egress rules, evaluated first-match-wins in list order.

CubeSandbox

Field Type Label Description
id string ID of the CubeSandbox.
namespace string containerd namespace.
created_at int64 Creation time of the sandbox.
containers Container repeated containers Sandbox.Spec.Containers
port_mappings PortMapping repeated Local cubevs portmapping
labels CubeSandbox.LabelsEntry repeated
numa_node int32
paused_at int64
private_cubebox_storage_data bytes cubebox_storage_data is the private data of the cubebox store. It is used to store the data of the cubebox store. this field only used with with_cubebox_store option for cubecli command.

CubeSandbox.LabelsEntry

Field Type Label Description
key string
value string

CubeSandboxFilter

PodSandboxFilter is used to filter a list of PodSandboxes.
All those fields are combined with 'AND'

Field Type Label Description
state ContainerStateValue State of the sandbox.
label_selector CubeSandboxFilter.LabelSelectorEntry repeated LabelSelector to select matches. Only api.MatchLabels is supported for now and the requirements are ANDed. MatchExpressions is not supported yet.
instance_type string There are some specific instance types for different scenarios. Each type has its own special logic during its lifecycle.

CubeSandboxFilter.LabelSelectorEntry

Field Type Label Description
key string
value string

DNSConfig

DNSConfig specifies the DNS servers and search domains of a sandbox.

Field Type Label Description
servers string repeated List of DNS servers of the cluster.
searches string repeated List of DNS search domains of the cluster.
options string repeated List of DNS options. See https://linux.die.net/man/5/resolv.conf for all available options.

DestroyCubeSandboxRequest

Field Type Label Description
requestID string requestID reqID
sandboxID string ID of the Sandbox.
annotations DestroyCubeSandboxRequest.AnnotationsEntry repeated Annotations can also be useful for runtime authors to experiment with new features that are opaque to the Kubernetes APIs (both user-facing and the CRI). Whenever possible, however, runtime authors SHOULD consider proposing new typed fields for any new features instead.
filter CubeSandboxFilter filter provide a strong assumption for destroy if not match, do nothing

DestroyCubeSandboxRequest.AnnotationsEntry

Field Type Label Description
key string
value string

DestroyCubeSandboxResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxID string ID of the Sandbox.
ext_info DestroyCubeSandboxResponse.ExtInfoEntry repeated

DestroyCubeSandboxResponse.ExtInfoEntry

Field Type Label Description
key string
value bytes

Device

Device specifies a host device to mount into a container.

Field Type Label Description
container_path string Path of the device within the container.
host_path string Path of the device on the host.
permissions string Cgroups permissions of the device, candidates are one or more of * r - allows container to read from the specified device. * w - allows container to write to the specified device. * m - allows container to create device files that do not yet exist.

EgressRule

Field Type Label Description
name string
match EgressRuleMatch optional
action EgressRuleAction optional

EgressRuleAction

Field Type Label Description
allow bool
audit string optional
inject EgressRuleInject repeated

EgressRuleInject

Field Type Label Description
header string
secret string
format string optional

EgressRuleMatch

Field Type Label Description
sni string optional
host string optional
method string repeated
path string optional
scheme string optional
port int32 optional L7 destination port. When set, scheme MUST also be set — together they pin the (host, port, scheme) tuple CubeEgress intercepts via skb->mark + iptables TPROXY. Both omitted keeps the legacy default {80/http, 443/https} behavior.

EmptyDirVolumeSource

EmptyDirVolumeSource represents an empty directory for a sandbox.

Field Type Label Description
Medium StorageMedium media more like a scheduling problem - user says what traits they need, we give them a backing store that satisfies that. For now this will cover the most common needs. Optional: what type of storage medium should back this directory. The default is "" which means to use the node's default medium. +optional
SizeLimit string Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a Sandbox. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir default 512Mi

ExecCubeSandboxRequest

Field Type Label Description
requestID string requestID reqID
sandbox_id string 容器所在的sandbox的id
container_id string 要执行的容器ID。
terminal bool 是否为交互式终端模式。
args string repeated Args specifies the binary and arguments for the application to execute.
env string repeated Env populates the process environment for the process.
cwd string Cwd is the current working directory for the process and must be relative to the container's root.

ExecCubeSandboxResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.

GetLocalSnapshotRequest

Field Type Label Description
requestID string requestID reqID
snapshotID string Logical snapshot ID to look up.

GetLocalSnapshotResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
snapshot LocalSnapshotInfo Catalog entry. Nil/empty when not found.

GetStorageMetricsRequest

Field Type Label Description
requestID string requestID reqID

GetStorageMetricsResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
node_id string Node identifier that produced this metric snapshot.
timestamp_unix_nano int64 Timestamp when metrics were collected.
metrics GetStorageMetricsResponse.MetricsEntry repeated Storage metrics reported by cubecow.

GetStorageMetricsResponse.MetricsEntry

Field Type Label Description
key string
value uint64

HTTPGetAction

HTTPGetAction describes an action based on HTTP Get requests.

Field Type Label Description
path string optional Path to access on the HTTP server.
port int32 The port to access on the container. Number must be in the range 1 to 65535.
host string optional Host name to connect to, defaults to the sandbox IP. You probably want to set "Host" in httpHeaders instead.
http_headers HTTPHeader repeated Custom headers to set in the request. HTTP allows repeated headers. +optional

HTTPHeader

HTTPHeader describes a custom header to be used in HTTP probes

Field Type Label Description
name string optional The header field name. This will be canonicalized upon output, so case-variant names will be understood as the same header.
value string optional The header field value

Hook

Field Type Label Description
path string
args string repeated
env string repeated
timeout int32 optional

Hooks

Field Type Label Description
Prestart Hook repeated

HostAlias

HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the
sandbox's hosts file.

Field Type Label Description
hostnames string repeated Hostnames for the above IP address.
ip string IP address of the host file entry.

HostDirSource

HostDirSource defines a single host directory to be shared into the VM/container
via a virtiofs propagation channel.

Field Type Label Description
name string Unique name for this source, referenced by VolumeMount.name.
host_path string Absolute path of the directory on the host to share.

HostDirVolumeSources

HostDirVolumeSources contains a list of host directory volumes to share into
containers via virtiofs propagation channels (virtio_ro / virtio_rw).

Field Type Label Description
volume_sources HostDirSource repeated List of host directory sources.

IDMapping

IDMapping describes host to container ID mappings for a pod sandbox.

Field Type Label Description
host_id uint32 HostId is the id on the host.
container_id uint32 ContainerId is the id in the container.
length uint32 Length is the size of the range to map.

InspectStorageVolumesRequest

Field Type Label Description
requestID string requestID reqID
bucket string Optional bucket name; defaults to "emptydir/v1" when empty.

InspectStorageVolumesResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxes SandboxStorageInfo repeated Sandbox storage records, after device-path refresh.

Int64Value

Int64Value is the wrapper of int64.

Field Type Label Description
value int64 The value.

KeyValue

Field Type Label Description
key string
value string

LifecycleHandler

LifecycleHandler defines a specific action that should be taken in a lifecycle hook. One and only one of the fields, except TCPSocket must be specified

Field Type Label Description
http_get HTTPGetAction

LinuxSeccompArg

Field Type Label Description
index uint32
value uint64
valueTwo uint64
op string op ref:LinuxSeccompOperator

ListCubeSandboxOption

Field Type Label Description
private_with_cubebox_store bool use cubebox store data warning: this private field only used by cubecli

ListCubeSandboxRequest

Field Type Label Description
id string optional ID of the sandbox.
filter CubeSandboxFilter optional CubeSandboxFilter to filter a list of CubeSandboxes.
option ListCubeSandboxOption optional

ListCubeSandboxResponse

Field Type Label Description
items CubeSandbox repeated List of CubeSandboxes.

ListLocalSnapshotsRequest

Field Type Label Description
requestID string requestID reqID

ListLocalSnapshotsResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
snapshots LocalSnapshotInfo repeated Snapshots known locally to this cubelet.

ListSandboxSnapshotsRequest

Field Type Label Description
requestID string requestID reqID
sandboxID string Logical sandbox ID for logging / correlation.
objects CowObjectRef repeated cubecow objects that should be inspected on this node.
meta_dir string Snapshot metadata directory that must remain restorable.

ListSandboxSnapshotsResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxID string Logical sandbox ID.
objects CowObjectStatus repeated Inspection results for requested cubecow objects.
meta_dir_exists bool Whether the snapshot metadata directory exists.
snapshot_state_path string Canonical snapshot state directory used for restore.
snapshot_state_exists bool Whether the snapshot state directory exists.
path_error_message string Path inspection error if any.

LocalSnapshotInfo

Field Type Label Description
snapshotID string Logical snapshot ID.
instance_type string Instance type (e.g. "cubebox").
spec_dir string Spec directory used to scope the snapshot (e.g. "2C2000M").
snapshot_path string Absolute on-disk snapshot path.
meta_dir string Snapshot metadata directory used for restore.
rootfs_vol string Authoritative cubecow rootfs object name.
rootfs_kind string Authoritative cubecow rootfs object kind.
memory_vol string Authoritative cubecow memory object name.
memory_kind string Authoritative cubecow memory object kind.
rootfs_size_bytes uint64 Rootfs size in bytes at snapshot time.
created_at string RFC3339 timestamp captured at commit time.
build_rootfs_vol string Build rootfs object name (template-build path only). Used to clean up the temporary writable layer during CleanupTemplate.
build_rootfs_kind string Build rootfs object kind.
kind string Catalog entry kind: "template" (AppSnapshot) or "runtime_snapshot" (CommitSandbox). Empty for pre-v4 legacy entries.

OCIConfig

Field Type Label Description
devices Device repeated Devices for the container.
cdi_devices CDIDevice repeated CDI devices for the container.

PingAction

TCPSocketAction describes an action based on ping.

Field Type Label Description
udp bool

PortMapping

Field Type Label Description
container_port int32 Port exposed in the container.
host_port int32 Port exposed on the host.

PostStop

Field Type Label Description
timeout_ms int32 Length of request timeout. In Millisecond.
lifecyle_handler LifecycleHandler How often (in Millisecond) to perform the probe. The action taken to execute the afterstop hook The action taken to determine the health of a container

PreStop

PreStop This hook is called immediately before a container is terminated due to an API request or management event such as a liveness/startup probe failure, preemption,
resource contention and others.

Field Type Label Description
termination_grace_period_ms int32 Length of time before container is killed. In Millisecond.
lifecyle_handler LifecycleHandler The action taken to execute the prestop hook The action taken to determine the health of a container

Probe

Probe describes a health check to be performed against a container to determine whether it is
alive or ready to receive traffic.

Field Type Label Description
probe_handler ProbeHandler The action taken to determine the health of a container
initial_delay_ms int32 Length of time before health checking is activated. In Millisecond. +optional
timeout_ms int32 Length of time before health checking times out. In Millisecond. +optional
period_ms int32 How often (in Millisecond) to perform the probe. +optional
success_threshold int32 Minimum consecutive successes for the probe to be considered successful after having failed. Must be 1 for liveness and startup.
failure_threshold int32 Minimum consecutive failures for the probe to be considered failed after having succeeded. +optional
probe_timeout_ms int32 单次探测超时时间(毫秒),默认100 +optional

ProbeHandler

ProbeHandler defines a specific action that should be taken in a probe.
One and only one of the fields must be specified.

Field Type Label Description
tcp_socket TCPSocketAction TCPSocket specifies an action involving a TCP port. +optional
ping PingAction Ping specifies the ping request to perform. +optional
http_get HTTPGetAction HTTPGet specifies the http request to perform. +optional

RLimit

Rlimits specifies rlimit options to apply to the process.

Field Type Label Description
no_file uint64 the RLIMIT_NOFILE value to set

Resource

Resource
ref by https://kubernetes.io/zh-cn/docs/reference/kubernetes-api/common-definitions/quantity/#Quantity

Field Type Label Description
cpu string Cpu的限制,单位为core数
mem string 内存限制,单位可以为字节
cpu_limit string mvm内对容器的cpu限制,单位为core数
mem_limit string mvm内对容器的内存限制,单位可以为字节

RollbackSandboxRequest

Field Type Label Description
requestID string requestID reqID
sandboxID string ID of the running sandbox to rollback.
snapshotID string Logical snapshot ID. When rootfs_vol/memory_vol/meta_dir are empty, cubelet resolves them from its local snapshot catalog keyed by this id.
rootfs_vol string Optional cubecow rootfs object name for the snapshot. Empty means "resolve from local catalog using snapshotID".
memory_vol string Optional cubecow memory object name for the snapshot. Empty means "resolve from local catalog using snapshotID".
meta_dir string Optional snapshot metadata directory. Empty means "resolve from local catalog using snapshotID".
new_gen uint32 New sandbox rootfs generation to derive.
desired_size uint64 Minimum rootfs size after deriving the new generation.

RollbackSandboxResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxID string ID of the sandbox that was rolled back.
snapshotID string Logical snapshot ID.
rootfs_vol string New sandbox rootfs volume name.
rootfs_kind string New sandbox rootfs kind.
rootfs_dev string Diagnostic-only rootfs device path; re-resolve from rootfs_vol on use.
new_gen uint32 New sandbox rootfs generation.
old_rootfs_vol string Old sandbox rootfs volume name, if known.
old_rootfs_deleted bool Whether deleting old_rootfs_vol succeeded.
memory_vol string Snapshot memory volume cubelet resolved during rollback. Master records it on the runtime ref so cleanup paths can detach the memory binding without re-fetching the local catalog.

RunCubeSandboxRequest

Field Type Label Description
requestID string request_ID reqID
volumes Volume repeated volumes
containers ContainerConfig repeated containers Sandbox.Spec.Containers
annotations RunCubeSandboxRequest.AnnotationsEntry repeated Annotations can also be useful for runtime authors to experiment with new features that are opaque to the Kubernetes APIs (both user-facing and the CRI). Whenever possible, however, runtime authors SHOULD consider proposing new typed fields for any new features instead.
labels RunCubeSandboxRequest.LabelsEntry repeated Key-value pairs that may be used to scope and select individual resources.
exposed_ports int64 repeated expose ports for external access
runtime_handler string Named runtime configuration to use for this PodSandbox. If the runtime handler is unknown, this request should be rejected. An empty string should select the default handler, equivalent to the behavior before this feature was added. See https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class
instance_type string There are some specific instance types for different scenarios. Each type has its own special logic during its lifecycle.
network_type string NetworkType default to tap
namespace string user define namespace
cube_network_config CubeNetworkConfig optional Egress network policy for the sandbox template/runtime.

RunCubeSandboxRequest.AnnotationsEntry

Field Type Label Description
key string
value string

RunCubeSandboxRequest.LabelsEntry

Field Type Label Description
key string
value string

RunCubeSandboxResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
sandboxID string ID of the Sandbox.
sandboxUUID string UUID of the Sandbox(mvm uuid).
sandboxIP string ip of the Sandbox(mvm ip).
ext_info RunCubeSandboxResponse.ExtInfoEntry repeated
port_mappings PortMapping repeated

RunCubeSandboxResponse.ExtInfoEntry

Field Type Label Description
key string
value bytes

SELinuxOption

SELinuxOption are the labels to be applied to the container.

Field Type Label Description
user string
role string
type string
level string

SandboxPathVolumeSource

Represents a sandbox path mapped into a container.

Field Type Label Description
path string path of the directory on the sandbox vm.
type string type for SandboxPath Volume

SandboxStorageInfo

SandboxStorageInfo aggregates every backend volume cubelet has registered
for a single sandbox.

Field Type Label Description
namespace string
sandboxID string
volumes StorageVolumeInfo repeated

StorageOrphanEntry

Field Type Label Description
format string Format root the file lives under (e.g. "512Mi").
file_path string Absolute path to the orphan file.
removed bool True when the file was successfully removed by this RPC. Always false for dry-run responses.
error_message string Removal error if any (only set when removed=false and dry_run=false).

StorageVolumeInfo

StorageVolumeInfo describes a single backend file/volume entry attached to a
sandbox, as understood by cubelet at the time of inspection.

Field Type Label Description
name string Logical mount name within the sandbox spec.
file_path string Filesystem path to mount inside the sandbox. For cubecow volumes this is re-resolved through the engine on every call.
size_limit uint64 Size limit in bytes (0 if not pre-allocated).
volume_name string cubecow object name backing this volume. Empty for legacy file-only entries that bypass cubecow.
kind string cubecow object kind ("volume"/"snapshot"). Empty for non-cubecow entries.
gen uint32 Generation number, used by rollback to derive successor objects.

SysCall

Field Type Label Description
names string repeated
action string action ref:LinuxSeccompAction
errno uint32
args LinuxSeccompArg repeated args ref:LinuxSeccompArg

TCPSocketAction

TCPSocketAction describes an action based on opening a socket.

Field Type Label Description
port int32
host string

UpdateCubeSandboxRequest

Field Type Label Description
requestID string requestID reqID
sandboxID string ID of the Sandbox.
annotations UpdateCubeSandboxRequest.AnnotationsEntry repeated Annotations can also be useful for runtime authors to experiment with new features that are opaque to the Kubernetes APIs (both user-facing and the CRI). Whenever possible, however, runtime authors SHOULD consider proposing new typed fields for any new features instead.

UpdateCubeSandboxRequest.AnnotationsEntry

Field Type Label Description
key string
value string

UpdateCubeSandboxResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
ext_info UpdateCubeSandboxResponse.ExtInfoEntry repeated Same shape as DestroyCubeSandboxResponse.ext_info. Pause returns volume ref-count events after in-process keep_tombstone cleanup.

UpdateCubeSandboxResponse.ExtInfoEntry

Field Type Label Description
key string
value bytes

Volume

Volume represents a named volume in a pod that may be accessed by any container in the pod.

Field Type Label Description
name string Required: This must be a DNS_LABEL. Each volume in a Sandbox must have a unique name. 这里必须包含用户资源的唯一标识,比如:tenant+funcid
volume_source VolumeSource The VolumeSource represents the location and type of a volume to mount.

VolumeMounts

VolumeMount describes a mounting of a Volume within a container.

Field Type Label Description
name string Required: This must be a DNS_LABEL. Each volume in a Sandbox must have a unique name.
container_path string Path of the mount within the container.
readonly bool If set, the mount is read-only.
propagation MountPropagation Requested propagation mode.
uid string UID for tmpfs mount (only applies to tmpfs volumes)
gid string GID for tmpfs mount (only applies to tmpfs volumes)
exec bool If set to true, allow execution on tmpfs mount (only applies to tmpfs volumes)
host_path string Path of the mount on the host. If the hostPath doesn't exist, then runtimes should report error. If the hostpath is a symbolic link, runtimes should follow the symlink and mount the real destination to container.
uidMappings IDMapping repeated Requested propagation mode. UidMappings specifies the runtime UID mappings for the mount.
gidMappings IDMapping repeated GidMappings specifies the runtime GID mappings for the mount.
recursive_read_only bool If set to true, the mount is made recursive read-only. In this CRI API, recursive_read_only is a plain true/false boolean, although its equivalent in the Kubernetes core API is a quaternary that can be nil, "Enabled", "IfPossible", or "Disabled". kubelet translates that quaternary value in the core API into a boolean in this CRI API. Remarks: - nil is just treated as false - when set to true, readonly must be explicitly set to true, and propagation must be PRIVATE (0). - (readonly == false && recursive_read_only == false) does not make the mount read-only.
selinux_relabel bool If set, the mount is read-only.
sub_path string Specific image sub path to be used from inside the image instead of its root, only necessary if the above image field is set. If the sub path is not empty and does not exist in the image, then runtimes should fail and return an error. Introduced in the Image Volume Source KEP beta graduation: https://kep.k8s.io/4639

VolumeSource

VolumeSource represents the source location of a volume to mount.
Only one of its members may be specified.

Field Type Label Description
empty_dir EmptyDirVolumeSource
sandbox_path SandboxPathVolumeSource
host_dir_volumes HostDirVolumeSources host_dir_volumes shares host directories into the container via virtiofs.
image cubelet.services.images.v1.ImageVolumeSource image volume source for image volume mount
plugin_volume cubelet.services.volumeplugin.v1.PluginVolumeSource plugin_volume delegates provisioning to a named external VolumePlugin (built-in, binary or RPC). All other fields above are handled by the existing storage pipeline and are NOT routed through the plugin framework.

ContainerState

ContainerState represents the lifecycle state of a container.

Name Number Description
CONTAINER_CREATED 0
CONTAINER_RUNNING 1
CONTAINER_EXITED 2
CONTAINER_UNKNOWN 3
CONTAINER_PAUSING 4
CONTAINER_PAUSED 5

InstanceType

Name Number Description
cubebox 0

MountPropagation

MountPropagation determines how mounts are propagated from the host
to container and the other way around.
When not set, MountPropagationNone is used.

Name Number Description
PROPAGATION_PRIVATE 0 No mount propagation ("private" in Linux terminology).
PROPAGATION_HOST_TO_CONTAINER 1 Mounts get propagated from the host to the container ("rslave" in Linux).
PROPAGATION_BIDIRECTIONAL 2 Mounts get propagated from the host to the container and from the container to the host ("rshared" in Linux).

NetworkType

Name Number Description
tap 0

SandboxPathType

Name Number Description
Directory 0 A directory must exist at the given path
Cgroup 1 A cgroup mount must exist at the given path
SharedBindMount 2

StorageMedium

Name Number Description
StorageMediumDefault 0 use blk device the default is for the node
StorageMediumMemory 1 use memory (e.g. tmpfs on linux)
StorageMediumCubeMsg 2 use cubemsg

CubeboxMgr

Service for handling cubesandbox

Method Name Request Type Response Type Description
Create RunCubeSandboxRequest RunCubeSandboxResponse
Destroy DestroyCubeSandboxRequest DestroyCubeSandboxResponse
List ListCubeSandboxRequest ListCubeSandboxResponse
Update UpdateCubeSandboxRequest UpdateCubeSandboxResponse
Exec ExecCubeSandboxRequest ExecCubeSandboxResponse
AppSnapshot AppSnapshotRequest AppSnapshotResponse AppSnapshot creates a cubebox, makes an app snapshot, and destroys the cubebox. Required annotations: - cube.master.appsnapshot.create: "true" - cube.master.appsnapshot.template.id: "<template_id>"
CommitSandbox CommitSandboxRequest CommitSandboxResponse CommitSandbox snapshots an existing running sandbox into a template snapshot.
RollbackSandbox RollbackSandboxRequest RollbackSandboxResponse RollbackSandbox restores a running sandbox to a committed snapshot.
CleanupTemplate CleanupTemplateRequest CleanupTemplateResponse CleanupTemplate removes local template snapshot/base data from this node.
ListSandboxSnapshots ListSandboxSnapshotsRequest ListSandboxSnapshotsResponse ListSandboxSnapshots inspects the requested snapshot objects on this node.
ListLocalSnapshots ListLocalSnapshotsRequest ListLocalSnapshotsResponse ListLocalSnapshots returns the full catalog of snapshots known to this node. It is intended for master discovery / inspection so master does not need to persist physical references (vol/dev/path/meta_dir) in its tables.
GetLocalSnapshot GetLocalSnapshotRequest GetLocalSnapshotResponse GetLocalSnapshot returns the physical catalog entry for a single snapshot by snapshot id. Returns PreConditionFailed when no local record exists.
GetStorageMetrics GetStorageMetricsRequest GetStorageMetricsResponse GetStorageMetrics returns node-local cubecow storage metrics.
InspectStorageVolumes InspectStorageVolumesRequest InspectStorageVolumesResponse InspectStorageVolumes lists every sandbox storage record cubelet owns and re-resolves cubecow device paths so cubecli can render an authoritative view without touching boltdb or the cubecow SDK directly.
CleanupOrphanStorageFiles CleanupOrphanStorageFilesRequest CleanupOrphanStorageFilesResponse CleanupOrphanStorageFiles scans configured emptydir format roots, drops files that have no live sandbox owner, and reports the action taken.


Top

api/services/cubehost/v1/hostimage.proto

ClientMessage

Field Type Label Description
id string
status ResponseStatus
type MessageType
hello string 客户端连接通知
forwardImageResponse ForwardImageResponse 下载操作响应
common string 通用消息
listInflightRequest ListInflightRequest 列出正在进行的请求
imageFsStats ImageFsStats 获取镜像文件系统信息

DNSConfig

DNSConfig specifies the DNS servers and search domains of a sandbox.

Field Type Label Description
servers string repeated List of DNS servers of the cluster.
searches string repeated List of DNS search domains of the cluster.
options string repeated List of DNS options. See https://linux.die.net/man/5/resolv.conf for all available options.

ForwardImageRequest

Field Type Label Description
image cubelet.types.ImageSpec Spec of the image.
auth cubelet.types.AuthConfig Authentication configuration for pulling the image.
sandbox_config PodSandboxConfig Config of the PodSandbox, which is used to pull image in PodSandbox context.

ForwardImageResponse

Field Type Label Description
image HostImage

HostImage

Basic information about a container image.

Field Type Label Description
id string ID of the image.
repo_tags string repeated Other names by which this image is known.
repo_digests string repeated Digests by which this image is known.
size uint64 Size of the image in bytes. Must be > 0.
spec cubelet.types.ImageSpec ImageSpec for image which includes annotations
descriptors google.protobuf.Any repeated all descriptors for this image
LayerMounts LayerMount repeated LayerMounts is a list of mounts for the image layers.
image_devs ImageDev image_devs is list of snapshots belong to erofs image device. A image can consist of the following components: 1. Composed solely of erofs layers 2. The erofs layer serves as the bottom layer, and the regular image layer serves as the upper layer 3. Ordinary image layer That is to say, the image cannot be composed of a regular image layer as the bottom layer and an erofs image layer as the upper layer. And a maximum of one erofs device can be used for one image.

ImageDev

Field Type Label Description
serial_id string
host_device_path string
vm_device_path string vm_device_path is the path of the device in the VM. this fileds is only used in the VM.
vm_root_path string this fileds is only used in the VM.
fs_type string fs_type is the filesystem type of the volume.
mount_options string repeated
snapshots string repeated snapshots is a list of snapshots of the image.
image_id string image id of all snapshot
image_references string repeated
bdf string pci bdf of virtio-blk

ImageFsStats

Field Type Label Description
AvailableBytes uint64
CapacityBytes uint64
UsedBytes uint64

InflightRequest

Field Type Label Description
id string

LayerMount

Mount specifies a host volume to mount into a container.

Field Type Label Description
name string
host_path string
vm_path string
digest string
parent string
usage string
dev_serial_id string dev_serial_id is the serial id of the device. this fileds is only used when the mount is a device.like erofs. When a layer belongs to a certain device, it is not allowed to use the host_path field again
layer_type LayerType

ListImagesRequest

Field Type Label Description
filter cubelet.types.ImageFilter Filter to list images.

ListImagesResponse

Field Type Label Description
images cubelet.types.Image repeated List of images.

ListInflightRequest

ListInflightRequest is the request message for listing inflight requests.

Field Type Label Description
requests InflightRequest repeated

MountImageRequest

Field Type Label Description
ref string
target_path string
error_msg string

PodSandboxConfig

PodSandboxConfig holds all the required and optional fields for creating a
sandbox.

Field Type Label Description
metadata PodSandboxMetadata Metadata of the sandbox. This information will uniquely identify the sandbox, and the runtime should leverage this to ensure correct operation. The runtime may also use this information to improve UX, such as by constructing a readable name.
hostname string Hostname of the sandbox. Hostname could only be empty when the pod network namespace is NODE.
log_directory string Path to the directory on the host in which container log files are stored. By default the log of a container going into the LogDirectory will be hooked up to STDOUT and STDERR. However, the LogDirectory may contain binary log files with structured logging data from the individual containers. For example, the files might be newline separated JSON structured logs, systemd-journald journal files, gRPC trace files, etc. E.g., PodSandboxConfig.LogDirectory = /var/log/pods/&lt;NAMESPACE&gt;_&lt;NAME&gt;_&lt;UID&gt;/ ContainerConfig.LogPath = containerName/Instance#.log
dns_config DNSConfig DNS config for the sandbox.
labels PodSandboxConfig.LabelsEntry repeated Port mappings for the sandbox. repeated PortMapping port_mappings = 5; Key-value pairs that may be used to scope and select individual resources.
annotations PodSandboxConfig.AnnotationsEntry repeated Unstructured key-value map that may be set by the kubelet to store and retrieve arbitrary metadata. This will include any annotations set on a pod through the Kubernetes API.

Annotations MUST NOT be altered by the runtime; the annotations stored here MUST be returned in the PodSandboxStatus associated with the pod this PodSandboxConfig creates.

In general, in order to preserve a well-defined interface between the kubelet and the container runtime, annotations SHOULD NOT influence runtime behaviour.

Annotations can also be useful for runtime authors to experiment with new features that are opaque to the Kubernetes APIs (both user-facing and the CRI). Whenever possible, however, runtime authors SHOULD consider proposing new typed fields for any new features instead.

Optional configurations specific to Linux hosts. LinuxPodSandboxConfig linux = 8; Optional configurations specific to Windows hosts. WindowsPodSandboxConfig windows = 9; |

PodSandboxConfig.AnnotationsEntry

Field Type Label Description
key string
value string

PodSandboxConfig.LabelsEntry

Field Type Label Description
key string
value string

PodSandboxMetadata

PodSandboxMetadata holds all necessary information for building the sandbox name.
The container runtime is encouraged to expose the metadata associated with the
PodSandbox in its user interface for better user experience. For example,
the runtime can construct a unique PodSandboxName based on the metadata.

Field Type Label Description
name string Pod name of the sandbox. Same as the pod name in the Pod ObjectMeta.
uid string Pod UID of the sandbox. Same as the pod UID in the Pod ObjectMeta.
namespace string Pod namespace of the sandbox. Same as the pod namespace in the Pod ObjectMeta.
attempt uint32 Attempt number of creating the sandbox. Default: 0.

RemoveSnapshotRequest

RemoveSnapshotRequest is the request message for removing a snapshot.

Field Type Label Description
LayerMounts LayerMount repeated LayerMounts is a list of mounts for the image layers.

ResponseStatus

Field Type Label Description
code ResponseStatusCode
message string

ServerMessage

Field Type Label Description
id string
status ResponseStatus
type MessageType
hello string 客户端连接通知
forwardImageRequest ForwardImageRequest 下载操作响应
removeSnapshotRequest RemoveSnapshotRequest 删除快照请求
listInflightRequest ListInflightRequest 列出正在进行的请求

LayerType

Name Number Description
LAYER_TYPE_FS 0
LAYER_TYPE_DEVICE 1

MessageType

Name Number Description
CLIENT_HELLO 0
IMAGE_FORWARD 1
REMOVE_SNAPSHOT 2
LIST_INFLIGHT_REQUEST 3
IMAGE_FS_STATS 4

ResponseStatusCode

Name Number Description
OK 0
ERROR 1

CubeHostImageService

Method Name Request Type Response Type Description
ReverseStreamForwardImage ClientMessage stream ServerMessage stream ReverseStreamForwardImage Reverse forwarding image download interface: 1. After launching the sandbox, the cubelet on the host first attempts to send a hello message indicating successful connection. The server inside the VM should establish a context representing this connection is ready. 2. When the server inside the VM receives a CRI image download request, it should send a reverse forwardImageRequest to the host to request image download. 3. After completing the image download on the host, it should proactively notify the VM with the image information based on the request ID.

Note: When the host has not actively connected to the server inside the VM, the server is considered invalid and image forwarding is prohibited. |
| ListImages | ListImagesRequest | ListImagesResponse | ListImages lists existing images. |

CubeVMImageService

CubeVMImageService is used by cube image converter service in vm.

Method Name Request Type Response Type Description
ListImages ListImagesRequest ListImagesResponse ListImages lists existing images.
PullImage ForwardImageRequest ForwardImageResponse PullImage pulls an image from the host.
Mount MountImageRequest MountImageRequest mount image to target path


Top

api/services/errorcode/v1/errorcode.proto

Ret

Field Type Label Description
ret_code ErrorCode
ret_msg string

ErrorCode

Name Number Description
OK 0
Unknown -1
Success 200
Conflict 130409
GrpcError 130500
ListContainerFailed 130504
ContainerNotFound 130505
RemoveContainerFailed 130506
CreateContainerFailed 130507
StartContainerFailed 130508
BindContainerFailed 130511
ConcurrentFailed 130513
DownloadCodeFailed 130514
DownloadLayerFailed 130516
DiskNotEnough 130519
DeployCodeTimeout 130524
DeployLayerTimeout 130525
DeployCodeLayerTimeout 130526
NoSpaceLeftOnDevice 130527
RunContainerTimeout 130539
CreateStorageFailed 130545
CreateImageFailed 130546
CreateNetworkFailed 130547
InitHostFailed 130548
SetCubeboxCgroupLimitFailed 130549
LoadContainerFromMetaDataFailed 130550
DeleteTaskFailed 130551
DeleteContainerFromMetaDataFailed 130552
UpdateLocalMetaDataFailed 130553
ResolveLocalSpecFailed 130554
NewContainerMetaDataFailed 130555
NewTaskFailed 130556
WaitTaskFailed 130557
StartTaskFailed 130558
DestroyStorageFailed 130559
CreateShimlogFailed 130560
DestroyShimlogFailed 130561
CreateCgroupFailed 130562
DestroyCgroupFailed 130563
DestroyNetworkFailed 130564
UpdateLocalImageSpecFailed 130565
CreateVolumeFailed 130566
HostDiskNotEnough 130567
PreConditionFailed 130568
DestroyImageFailed 130569
TaskPauseFailed 130588
TaskResumeFailed 130589
InitCommandPathError 130445
ContainerStateExitedByUser 130451
PullImageFailed 130456
PullImageTimeOut 130457
PortBindingFailed 130459
PullImageTokenAuthFailed 130460
SquashfsMountFailed 130461
LoadUserProcTimeout 130462
ConcurrentLimited 130480
PreStopTimeOut 130481
PreStopFailed 130482
InvalidParamFormat 130483
TaskStateInvalid 130490
ExecCommandInSandboxFailed 130497 尝试在沙箱中执行命令失败
AppSnapshotNotExist 130498 应用快照不存在


Top

api/services/images/v1/images.proto

CreateImageRequest

Field Type Label Description
requestID string requestID reqID
spec ImageSpec

CreateImageRequestResponse

CreateImageRequestResponse contains the response for CreateImageRequest

Field Type Label Description
requestID string Unique identifier matching the request
ret cubelet.services.errorcode.v1.Ret Result of the operation

CubeListImageRequest

Field Type Label Description
requestID string requestID reqID
filter cubelet.types.ImageFilter Filter to list images.

CubeListImageResponse

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
images cubelet.types.Image repeated List of images.

DestroyImageRequest

DestroyImageRequest destroy image

Field Type Label Description
requestID string requestID reqID
spec ImageSpec

DestroyImageResponse

DestroyImageResponse destroy image

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.

ImageSpec

ImageSpec is an internal representation of an image.
ImageSpec defines the specification of an image

Field Type Label Description
image string Container's Image field (e.g. imageID or imageDigest).
annotations ImageSpec.AnnotationsEntry repeated annotations 传递username和token信息 ie. annotations["cube.image.username"] = xxx ie. annotations["cube.image.token"] = xxx
storage_media string Type of storage media for the image (docker/ext4)
namespace string
runtime_handler string

ImageSpec.AnnotationsEntry

Field Type Label Description
key string
value string

ImageVolumeSource

ImageVolumeSource represents a image volume resource.

Field Type Label Description
reference ImageSpec Required: Image or artifact reference to be used.
pullPolicy PullPolicy

VolumUtilsRequest

VolumUtilsRequest volum utils

Field Type Label Description
requestID string requestID reqID
cmd string cmd
input string input

VolumUtilsResponse

VolumUtilsResponse volum utils

Field Type Label Description
requestID string requestID reqID
ret cubelet.services.errorcode.v1.Ret Ret.
output string output

ImageStorageMediaType

Name Number Description
docker 0 Docker means normal oci container images.should be pulled from dockerhub
ext4 1 image from local disk in ext4 format rootfs.

PullPolicy

Policy for pulling OCI objects. Possible values are:
Always: the kubelet always attempts to pull the reference. Container creation will fail If the pull fails.
Never: the kubelet never pulls the reference and only uses a local image or artifact. Container creation will fail if the reference isn't present.
IfNotPresent: the kubelet pulls if the reference isn't already present on disk. Container creation will fail if the reference isn't present and the pull fails.
Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.
+optional

Name Number Description
Always 0 PullAlways means that kubelet always attempts to pull the latest image. Container will fail If the pull fails.
Never 1 PullNever means that kubelet never pulls an image, but only uses a local image. Container will fail if the image isn't present
IfNotPresent 2 PullIfNotPresent means that kubelet pulls if the image isn't present on disk. Container will fail if the image isn't present and the pull fails.

Images

Service for handling image operations.

Method Name Request Type Response Type Description
CreateImage CreateImageRequest CreateImageRequestResponse
ListImages CubeListImageRequest CubeListImageResponse cubelet v1 not support list and destroy image
DestroyImage DestroyImageRequest DestroyImageResponse
VolumeUtils VolumUtilsRequest VolumUtilsResponse


Top

api/services/multimetadb/v1/multimeta_db.proto

BucketDefine

bucket define for boltdb

Field Type Label Description
name string bucket 的名字
is_namespace bool
db_name string 当使用独立db时,需要指定db的名字。例如cgroup插件使用了独立db
describe string 对该bucket的描述

CommonRequestHeader

Field Type Label Description
requestID string

CommonResponseHeader

Field Type Label Description
requestID string
code cubelet.services.errorcode.v1.ErrorCode
Message string

DbData

Field Type Label Description
key string key of this data
buckets string repeated bucket 的定义,可能如 erofs {namespace} 这种二级定义
value bytes 数据的value

GetBucketDefinesResponse

Field Type Label Description
header CommonResponseHeader
bucket_defines BucketDefine repeated

GetDataRequest

Field Type Label Description
header CommonRequestHeader
buckets string repeated bucket 的定义,可能如 erofs {namespace} 这种二级定义
key string 数据的key
db_name string 当使用独立db时,需要指定db的名字。例如cgroup插件使用了独立db

MultiMetaDBServer

Service for machine level operations.

Method Name Request Type Response Type Description
GetBucketDefines CommonRequestHeader GetBucketDefinesResponse
GetStreamData GetDataRequest DbData stream 使用流式接口,批量查询db中的数据


Top

api/services/nbi/v1/cubelet_api.proto

InitRequest

Field Type Label Description
requestID string

InitResponse

Field Type Label Description
requestID string
code cubelet.services.errorcode.v1.ErrorCode
Message string
ext_info InitResponse.ExtInfoEntry repeated

InitResponse.ExtInfoEntry

Field Type Label Description
key string
value bytes

CubeLet

Service for machine level operations.

Method Name Request Type Response Type Description
InitHost InitRequest InitResponse Initialize the host, destroy all of the container and initialize metadata. This is a dangerous operation.


Top

api/services/version/v1/version.proto

VersionResponse

Field Type Label Description
version string
revision string

Version

Method Name Request Type Response Type Description
Version .google.protobuf.Empty VersionResponse


Top

api/services/volumeplugin/v1/volumeplugin.proto

AttachRequest

Field Type Label Description
sandbox_id string sandbox_id is the sandbox being created.
namespace string namespace is the containerd namespace.
volume_id string volume_id is the CubeMaster VolumeRecord identifier. Used as the ref-count DB key for cross-sandbox deduplication.
volume_base_dir string volume_base_dir is the parent directory that Cubelet requires the plugin to mount this volume under. The plugin MUST return a host_path located inside this directory (typically "<volume_base_dir>/<plugin>-<volume_id>"). Cubelet rejects the attach if host_path is not within volume_base_dir.
ref_count int64 ref_count is the number of sandboxes already attached to this volume BEFORE this call (i.e. the pre-attach count). 0 → first attach; plugin should perform host-level setup.
private_data string private_data is opaque plugin state returned by Create and persisted in t_cube_volume. CubeMaster forwards it on sandbox create so Attach can reuse Create-time context without a second control-plane round-trip. Max length: 1024 bytes. May be empty.

AttachResponse

Field Type Label Description
host_path string host_path is the host-side path cubelet exposes into the VM.
metadata AttachResponse.MetadataEntry repeated metadata is opaque key-value data cubelet persists and passes back unchanged to Detach.

AttachResponse.MetadataEntry

Field Type Label Description
key string
value string

CreateRequest

Field Type Label Description
volume_id string volume_id is the stable identifier assigned by CubeMaster.
name string name is the human-readable label.

CreateResponse

Field Type Label Description
token string token is an optional credential for volume-content access. volume_id and name come from the CreateRequest; plugins must not echo them.
private_data string private_data is opaque plugin state persisted in t_cube_volume and forwarded to Attach on sandbox create. Not returned to API/SDK clients. Max length: 1024 bytes. May be empty.

DestroyRequest

Field Type Label Description
volume_id string

DestroyResponse

DetachRequest

Field Type Label Description
sandbox_id string
namespace string
volume_id string
metadata DetachRequest.MetadataEntry repeated metadata is the exact map returned by the corresponding Attach call.
ref_count int64 ref_count is the number of sandboxes still attached AFTER this call (i.e. the post-detach count). 0 → last detach; plugin should tear down host-level resources.

DetachRequest.MetadataEntry

Field Type Label Description
key string
value string

DetachResponse

PluginVolumeSource

PluginVolumeSource selects an external volume plugin by driver name.
Embedded in cubebox.VolumeSource.

Field Type Label Description
driver string driver is the registered plugin name (e.g. "nfs", "s3fuse", "cfs"). Must match VolumePlugin.Name() on the Cubelet node.

VolumeControllerService

VolumeControllerService is the gRPC interface for CubeMaster Controller hooks.
RPC-type plugins implement this service alongside VolumePluginService (Node).

Method Name Request Type Response Type Description
Create CreateRequest CreateResponse Create provisions a new volume in the backend storage.
Destroy DestroyRequest DestroyResponse Destroy permanently deletes a volume and its backend data.

VolumePluginService

VolumePluginService is the gRPC interface implemented by RPC-type plugins.
The Cubelet VolumePlugin framework calls these RPCs instead of forking a
subprocess.

Method Name Request Type Response Type Description
Attach AttachRequest AttachResponse Attach provisions the volume on the host and returns attach metadata. Cubelet stores the returned metadata and passes it back unchanged on Detach.
Detach DetachRequest DetachResponse Detach tears down a previously attached volume.


Top

api/types/v1/descriptor.proto

Descriptor

Descriptor describes a blob in a content store.

This descriptor can be used to reference content from an
oci descriptor found in a manifest.
See https://godoc.org/github.com/opencontainers/image-spec/specs-go/v1#Descriptor

Field Type Label Description
media_type string
digest string
size int64
annotations Descriptor.AnnotationsEntry repeated
data google.protobuf.Any Any is a message type wrapper allowing you to attach arbitrary metadata to a blob.

Descriptor.AnnotationsEntry

Field Type Label Description
key string
value string


Top

api/types/v1/images.proto

AuthConfig

AuthConfig contains authorization information for connecting to a registry.

Field Type Label Description
username string
password string
auth string
serverAddress string
identityToken string IdentityToken is used to authenticate the user and get an access token for the registry.
registryToken string RegistryToken is a bearer token to be sent to a registry

Image

Basic information about a container image.

Field Type Label Description
id string ID of the image.
repoTags string repeated Other names by which this image is known.
repoDigests string repeated Digests by which this image is known.
size uint64 Size of the image in bytes. Must be > 0.
uid Int64Value UID that will run the command(s). This is used as a default if no user is specified when creating the container. UID and the following user name are mutually exclusive.
username string User name that will run the command(s). This is used if UID is not set and no user is specified when creating container.
spec ImageSpec ImageSpec for image which includes annotations
pinned bool Recommendation on whether this image should be exempt from garbage collection. It must only be treated as a recommendation -- the client can still request that the image be deleted, and the runtime must oblige.
namespace string namespace of image, this field is used for cube image management.

ImageFilter

Field Type Label Description
image ImageSpec Spec of the image.

ImageSpec

ImageSpec is an internal representation of an image.

Field Type Label Description
image string Container's Image field (e.g. imageID or imageDigest).
annotations ImageSpec.AnnotationsEntry repeated Unstructured key-value map holding arbitrary metadata. ImageSpec Annotations can be used to help the runtime target specific images in multi-arch images.

ImageSpec.AnnotationsEntry

Field Type Label Description
key string
value string

Int64Value

Int64Value is the wrapper of int64.

Field Type Label Description
value int64 The value.

Scalar Value Types

.proto Type Notes C++ Java Python Go C# PHP Ruby
double double double float float64 double float Float
float float float float float32 float float Float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int int32 int integer Bignum or Fixnum (as required)
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long int64 long integer/string Bignum
uint32 Uses variable-length encoding. uint32 int int/long uint32 uint integer Bignum or Fixnum (as required)
uint64 Uses variable-length encoding. uint64 long int/long uint64 ulong integer/string Bignum or Fixnum (as required)
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int int32 int integer Bignum or Fixnum (as required)
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long int64 long integer/string Bignum
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int uint32 uint integer Bignum or Fixnum (as required)
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long uint64 ulong integer/string Bignum
sfixed32 Always four bytes. int32 int int int32 int integer Bignum or Fixnum (as required)
sfixed64 Always eight bytes. int64 long int/long int64 long integer/string Bignum
bool bool boolean boolean bool bool boolean TrueClass/FalseClass
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode string string string String (UTF-8)
bytes May contain any arbitrary sequence of bytes. string ByteString str []byte ByteString string String (ASCII-8BIT)

File: Cubelet/pkg/store/membolt/README.md

BoltCacheStore - 泛型缓存存储

一个高性能的泛型缓存存储实现,结合了 Kubernetes cache.Store 的内存缓存和 BoltDB 的持久化存储。

特性

  • 🚀 高性能:O(1) 读取,支持并发访问
  • 💾 持久化:数据自动保存到 BoltDB
  • 🔄 自动同步:缓存更新时数据库自动同步
  • 🔒 线程安全:使用 RWMutex 保护并发访问
  • 📦 泛型支持:支持任意可序列化的类型
  • 🛡️ 容错能力:支持从数据库恢复缓存

快速开始

安装

bash
go get github.com/tencentcloud/CubeSandbox/Cubelet/pkg/store/membolt

基本使用

go
package main

import (
    "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/utils"
    "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/store/membolt"
    "k8s.io/client-go/tools/cache"
)

type User struct {
    ID   string `json:"id"`
    Name string `json:"name"`
}

func main() {
    // 初始化数据库
    db, _ := utils.NewCubeStore("/tmp/test.db", nil)
    defer db.Close()

    // 创建存储
    store := membolt.NewBoltCacheStore[User](
        db,
        func(obj interface{}) (string, error) {
            return obj.(User).ID, nil
        },
        cache.Indexers{},
    )

    // 添加对象
    store.Add(User{ID: "1", Name: "Alice"})

    // 查询对象
    item, exists, _ := store.GetByKey("1")
    if exists {
        user := item.(User)
        println(user.Name) // 输出: Alice
    }

    // 更新对象
    store.Update(User{ID: "1", Name: "Alice Updated"})

    // 删除对象
    store.Delete(User{ID: "1", Name: "Alice Updated"})
}

API 文档

构造函数

go
func NewBoltCacheStore[Obj any](
    db *utils.CubeStore,
    keyFunc cache.KeyFunc,
    indexers cache.Indexers,
) *BoltCacheStore[Obj]

创建一个新的 BoltCacheStore 实例。

参数:

  • db: BoltDB 数据库实例
  • keyFunc: 用于提取对象键的函数
  • indexers: 索引器配置

写操作

Add

go
func (bcs *BoltCacheStore[Obj]) Add(obj Obj) error

添加对象到缓存和数据库。

Update

go
func (bcs *BoltCacheStore[Obj]) Update(obj Obj) error

更新对象在缓存和数据库中的值。

Delete

go
func (bcs *BoltCacheStore[Obj]) Delete(obj Obj) error

从缓存和数据库中删除对象。

读操作

Get

go
func (bcs *BoltCacheStore[Obj]) Get(obj Obj) (item interface{}, exists bool, err error)

从缓存中获取对象。

GetByKey

go
func (bcs *BoltCacheStore[Obj]) GetByKey(key string) (item interface{}, exists bool, err error)

通过键从缓存中获取对象。

List

go
func (bcs *BoltCacheStore[Obj]) List() []interface{}

列出缓存中的所有对象。

ListKeys

go
func (bcs *BoltCacheStore[Obj]) ListKeys() []string

列出缓存中的所有键。

高级操作

Replace

go
func (bcs *BoltCacheStore[Obj]) Replace(list []interface{}, resourceVersion string) error

替换缓存和数据库中的所有对象。

Resync

go
func (bcs *BoltCacheStore[Obj]) Resync() error

从数据库恢复缓存数据。

工具方法

GetStore

go
func (bcs *BoltCacheStore[Obj]) GetStore() cache.Store

返回底层的 cache.Store 实例。

GetDatabase

go
func (bcs *BoltCacheStore[Obj]) GetDatabase() *utils.CubeStore

返回底层的数据库实例。

数据库 Bucket 命名

BoltCacheStore 自动为每个类型创建一个 bucket,命名规则为:

text
generic_<TypeName>

例如:

  • User 类型 → generic_User
  • Pod 类型 → generic_Pod
  • Service 类型 → generic_Service

并发安全

BoltCacheStore 使用 sync.RWMutex 保护所有操作:

  • 读操作(并发):Get, GetByKey, List, ListKeys
  • 写操作(互斥):Add, Update, Delete, Replace, Resync

错误处理

所有操作都返回 error,可能的错误包括:

  • "failed to get key: %w" - KeyFunc 错误
  • "failed to marshal object: %w" - 序列化错误
  • "failed to add to cache: %w" - 缓存操作错误
  • "failed to set in database: %w" - 数据库操作错误

性能特性

操作 时间复杂度 说明
Add O(n) n = 序列化大小
Update O(n) n = 序列化大小
Delete O(1) 键查询
Get O(1) 键查询
GetByKey O(1) 键查询
List O(m) m = 缓存中的对象数
ListKeys O(m) m = 缓存中的对象数
Replace O(n*m) n = 新对象数,m = 序列化大小
Resync O(n*m) n = 数据库中的对象数,m = 序列化大小

测试

运行所有测试

bash
go test -v ./pkg/store/membolt/

运行基准测试

bash
go test -bench=. ./pkg/store/membolt/

测试覆盖

  • ✅ Add 操作
  • ✅ Update 操作
  • ✅ Delete 操作
  • ✅ Get/GetByKey 查询
  • ✅ List/ListKeys 列表
  • ✅ Replace 替换
  • ✅ Resync 同步
  • ✅ 并发访问
  • ✅ 错误处理

文档

最佳实践

  1. KeyFunc 设计

    go
    keyFunc := func(obj interface{}) (string, error) {
        user := obj.(User)
        if user.ID == "" {
            return "", fmt.Errorf("empty user ID")
        }
        return user.ID, nil
    }
  2. 错误处理

    go
    if err := store.Add(obj); err != nil {
        log.Printf("Failed to add object: %v", err)
        return err
    }
  3. 资源管理

    go
    db, err := utils.NewCubeStore(path, nil)
    if err != nil {
        return err
    }
    defer db.Close()
  4. 并发访问

    go
    // 充分利用读锁的并发性能
    items := store.List()
    for _, item := range items {
        // 处理 item
    }
  5. 数据一致性

    go
    // 定期验证一致性
    if err := store.Resync(); err != nil {
        log.Printf("Resync failed: %v", err)
    }

已知限制

  1. 序列化开销:每次操作都需要 JSON 序列化/反序列化
  2. 数据库 I/O:每次写操作都会触发数据库写入
  3. 内存占用:缓存会占用内存,大数据集需要考虑内存限制
  4. 类型名称:依赖 reflect.TypeOf().Name(),某些情况下可能为空

常见问题

Q: 如何处理缓存不一致?
A: 调用 store.Resync() 从数据库恢复缓存

Q: 支持哪些类型?
A: 任何可以被 JSON 序列化的类型

Q: 是否线程安全?
A: 是的,使用 RWMutex 保护并发访问

Q: 性能如何?
A: 读操作 O(1),写操作 O(n)(n = 序列化大小)

Q: 如何自定义 KeyFunc?
A: 在创建存储时传入自定义的 KeyFunc 函数

许可证

MIT

贡献

欢迎提交 Issue 和 Pull Request!

相关资源