{"owner":"psviderski","repo":"uncloud","hasSkills":true,"totalSkillsCount":2,"totalTokensCount":4106,"categories":["project-spec","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["misc/design.md","AGENTS.md"],"skills":{"misc/design.md":"# Uncloud design\n\nUncloud aims to provide a solution for deploying and running containerized applications and services across various\nplatforms — from cloud VPSs to Raspberry Pis and powerful servers — or any combination of them.\n\nWhile existing prominent PaaS solutions like https://coolify.io/ and https://dokploy.com/ use a single machine or Docker\nSwarm as their underlying infrastructure, Uncloud takes a different approach. It aims to offer a multi-host experience\nsimilar to the serverless solutions provided by cloud providers but on users' own machines. Users should be able to add\nmachines as compute resources without worrying about control plane high availability and cluster management. For\nexample, a user could combine a cloud VPS with a spare Raspberry Pi to create a unified computing environment.\n\nUncloud aims to replace the concept of a \"cluster\" with a \"network\" of machines. It's similar to Tailscale's design\nwhere users can add machines to a network to enable secure communication between them. The primary challenges in this\ndesign are:\n\n1. Building such a network\n2. Orchestrating containers across it\n3. Exposing services to the internet\n\n## Network of machines\n\nWhile Tailscale is an obvious solution for establishing an overlay network, it has some drawbacks:\n\n1. Users must register an account and generate a key for each machine.\n2. Every container requires a Tailscale proxy to communicate with other machines and containers.\n\nThe Tailscale subnet router feature could potentially be used to route traffic from the machine to the internal Docker\nnetwork.\n\nAs an alternative, we can configure a simple flat WireGuard mesh network during the machine setup process. For example:\n\n| CIDR | Description |\n|-----------------|----------------------------------------------------------|\n| `10.210.0.0/16` | The entire WireGuard mesh network |\n| `10.210.X.0/24` | `/24` subnet assigned to machine `X` |\n| `10.210.X.1/32` | Machine `X` address is the first address from the subnet |\n| `10.210.X.Y/32` | Container `Y` address running on machine `X` |\n\nTesting has shown that inter-container and machine routing works well when `/24` is assigned to the Docker network\nbridge, while the WireGuard interface is configured with just the `/32` machine address.\n\nThis setup allows containers to communicate with each other on the same machine or across machines without address\ntranslation. This design also enables the future implementation of [ACLs](https://tailscale.com/kb/1018/acls) and\nsecurity groups to restrict traffic between machines and containers if needed.\n\nThe peer discovery and NAT traversal techniques for the WireGuard mesh are heavily inspired by the\nTalos [KubeSpan](https://www.talos.dev/v1.7/talos-guides/network/kubespan/) design.\n\n## Orchestration\n\nThe main question that drives the design of the orchestration system is can we build a system that doesn't require a\ncentralised control plane? The short answer is yes, we can. However, it's not easy as decentralised systems are\ninherently more complex than centralised ones.\n\nThe existing solutions such as Kubernetes, Docker Swarm, and Nomad use a centralised control plane to provide a single\npoint of entry for the user to interact with the cluster. I'm envisioning a system where all machines are equal, yet\neach one can represent the entire cluster and receive commands from the user. Where a user can be a CLI tool or a web\ninterface that can run on any machine in the network. In case of network partitioning, the user should continue to be\nable to interact with each partition separately to manage services running on them. The system should be able to\nreconcile the state of the network when the partition is healed. Such an architecture favors Availability and Partition\ntolerance (AP) over Consistency and Availability (CA) in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem).\n\n### Shared state\n\nOne of the ways to achieve this is to store the entire shared state on every machine and keep them all in sync. A couple\nof techniques can be used to assist with this:\na [Conflict-Free Replicated Data Type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) (CRDT) and a\npeer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol).\n\nBy composing the state using CRDT data structures allows any machine to modify its own copy of the state independently,\nconcurrently, and without coordination with other machines. The CRDT automatically resolves conflicts that might arise\nfrom concurrent updates. Although the machines may have different states at any given time, they're guaranteed to\neventually converge to the *same* state. The tradeoff is that the *same* state doesn't always mean the intended one from\neach user's point of view.\n\nThere is a distributed key-value store implementation in Go ([ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt)) from\nthe IPFS ecosystem that uses\n[Merkle CRDTs](https://research.protocol.ai/publications/merkle-crdts-merkle-dags-meet-crdts/psaras2020.pdf). It's\nindependent of the permanent storage implementation and the transport layer for broadcasting and receiving updates from\npeers.\n\nThe gossip protocol can be used to propagate state updates across the network. There is a great project from HashiCorp\ncalled [Serf](https://www.serf.io/) that provides a gossip protocol implementation in addition to cluster membership,\nfailure detection, and basic orchestration that is decentralized, fault-tolerant and highly available. It's used in\nNomad and Consul internally.\n\n[`experiment/serf_crdt.go`](../experiment/serf_crdt.go) is an experimental distributed key-value store\nusing [BadgerDB](https://github.com/dgraph-io/badger) as the persistent storage for\n[ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) and Serf as the gossip protocol that uses its custom user events\nand queries features. The implementation is very inefficient but the results are promising. A couple of machines are\nable to replicate changes to the store across the internet in a fraction of a second and converge their copies if they\nget out of sync. Although, the convergence is quite slow at the moment due to the inefficient broadcast and query\nimplementation.\n\nThere is a hypothesis that in practice there shouldn't be much churn and conflicts in the state of relatively small\ndeployments. The updates should be propagated quickly enough thus the eventually consistent behaviour should be\nacceptable.\n\n### Container scheduling\n\nThere are two fundamental approaches to design an orchestration system: declarative and imperative. I believe that\nUncloud should use a hybrid approach that favours the imperative over the declarative one where possible.\n\nThe imperative approach allows the errors to be handled in a more predictable way. While the declarative approach allows\nto decouple components of the system by using a shared state and events for communication.\n\nFor example, the user can run a command to start a container on a specific machine. The command can call the target\nmachine directly to start the container, handle the errors accordingly, and return the result to the user.\nAlternatively, the command can update the local state declaring that a container should be started on a specific machine\nand wait until it's propagated to the target machine and reconciled. Arguably, the latter approach is more complex, less\npredictable, and has more edge cases to handle.\n\nHowever, asynchronously updating the configuration of DNS servers or reverse proxies in accordance to the state changes\ncaused by started or stopped containers is likely a more reliable approach than doing it imperatively.\n\nNot sure if we really need this yet, but more complex scenarios like running and watching a replica set of containers\nmight require a sort of coordination between machines. I believe this can be achieved by using consensus and leader\nelection algorithms on demand.\n\nMaybe for now we should only aim for a simpler and more static container orchestrator where container scheduling can\nonly be initiated by a user. Docker on each machine will ensure that the containers are running and restarted on\nfailures, of course. But they won't be moved to other machines automatically.\n\n## Service discovery\n\nService discovery using DNS has its own drawbacks but it's likely the simplest way to implement it. Like in Docker and\nDocker Swarm, the Uncloud agent on every machine will expose an internal DNS server to the machine itself and the\ncontainers running on it. The DNS server will resolve machine, container, and service names to their respective IP\naddresses within the mesh network.\n\nDNS servers will watch running containers through the shared state and update their DNS records accordingly. The\nfollowing table is an example list of available DNS names:\n\n| DNS name | Resolves to |\n|----------------------------------------------------|----------------------------------------------------------------------------|\n| `.machine.internal` | Mesh IP of the machine. Machine names must be unique within the org |\n| `..machine.internal` | Mesh IP of the container on the machine |\n| `.internal` | Mesh IPs of all containers for the service |\n| `lb..internal` | Virtual IP / LB IP that balances traffic to all containers for the service |\n\nA proper thinking about the name design is needed. Most likely we need to namespace them by project or environment name\nto allow for multiple instances of the same service and avoid conflicts.\n\nVirtual IPs for services using IPVS, ip/nftables, or eBPF can be implemented later if we see fit.\n\n## Ingress\n\nServices can be exposed to the internet by running a reverse proxy on a machine(s) that listens on the public IP\naddress. A user should be able to control which machine(s) should run a reverse proxy by assigning an appropriate role\nto them.\n\nTraefik or Caddy can be used as a reverse proxy that can automatically discover services running on the network using\nthe internal DNS server. They both support automatic TLS certificate generation and renewal using Let's Encrypt.\n\nIn case of network partitioning, DNS servers should adjust their records in accordance with what containers are\navailable in their partition. Consequently, the reverse proxy should adjust its configuration to route traffic to only\nthe available containers by resolving the updated DNS records.\n","AGENTS.md":"# AGENTS.md - Uncloud Project Guide\n\nThis document provides comprehensive information about the Uncloud project for AI assistants to understand the codebase,\narchitecture, and development practices.\n\n## Project Overview\n\n**Uncloud** is a lightweight clustering and container orchestration tool that enables deployment and management of web\napplications across cloud VMs and bare metal servers. It creates a secure WireGuard mesh network between Docker hosts\nand provides automatic service discovery, load balancing, HTTPS ingress, and simple CLI commands for application\nmanagement.\n\n### Key Characteristics\n\n- **Language**: Go\n- **Architecture**: Decentralized, no control plane\n- **Target**: Self-hosted infrastructure without Kubernetes complexity\n- **License**: View LICENSE file for details\n- **Status**: Active development, not yet ready for production\n\n## Core Features\n\n### 🏗️ Infrastructure\n\n- **Multi-machine deployment**: Combine cloud VMs, dedicated servers, and bare metal\n- **Zero-config networking**: Automatic WireGuard mesh with NAT traversal\n- **Decentralized design**: No central control plane, all machines are equal\n- **Service discovery**: Built-in DNS server resolves service names to container IPs\n\n### 🚀 Application Management\n\n- **Docker Compose compatibility**: Uses familiar Docker Compose format\n- **Zero-downtime deployments**: Rolling updates without service interruption\n- **Automatic HTTPS**: Caddy reverse proxy with Let's Encrypt integration\n- **Managed DNS**: Free `*.xxxxxx.uncld.dev` subdomains via Uncloud DNS service\n- **Cross-machine scaling**: Run containers across multiple machines\n\n### 🔧 Developer Experience\n\n- **Docker-like CLI**: Familiar commands (`uc` binary)\n- **Imperative operations**: Direct commands vs. declarative state reconciliation\n- **Remote management**: Control entire infrastructure via SSH to any machine\n- **Minimal overhead**: ~150MB RAM footprint per machine\n\n## Architecture\n\n### Core Components\n\n1. **CLI (`uc`)** - Main user interface for cluster management\n2. **Daemon (`uncloudd`)** - Machine daemon running on each node\n3. **Corrosion** - Distributed SQLite database for cluster state (Fly.io project)\n4. **Caddy** - Reverse proxy for HTTPS termination and routing\n5. **WireGuard** - Secure mesh networking between machines\n\n### Network Architecture\n\n- Each machine gets unique subnet (e.g., `10.210.0.0/24`, `10.210.1.0/24`)\n- Containers get cluster-unique IPs for direct communication\n- Automatic peer discovery and key management\n- NAT traversal for machines behind firewalls\n\n### State Management\n\n- **CRDT-based distributed storage** using Corrosion\n- **Eventually consistent** state across all machines\n- **Gossip protocol** (Serf) for state propagation\n- **No quorum requirements** - partial network splits remain functional\n\n## Project Structure\n\n### Key Directories\n\n- **`cmd/`**: Contains main applications\n\n - `uc/`: CLI tool with subcommands for machine, service, volume management\n - `uncloudd/`: Daemon that runs on each machine\n - `ucind/`: Development cluster management for testing\n\n- **`internal/`**: Internal implementation packages\n\n - `cli/`: Command-line interface logic\n - `machine/`: Machine lifecycle and state management\n - `daemon/`: Daemon implementation and gRPC services\n - `dns/`: Internal DNS server for service discovery\n\n- **`pkg/`**: Public API packages for external use\n\n - `api/`: Core API types and definitions\n - `client/`: Client libraries for interacting with Uncloud\n\n- **`experiment/`**: Experimental features and prototypes\n- **`scripts/`**: Installation and utility scripts\n- **`test/`**: Test suites and test infrastructure\n- **`website/`**: Documentation website (Docusaurus)\n\n - `landing/`: Landing page\n - `docs/`: User documentation\n\n- **`misc/`**: Design documents and guides\n\n## Key Technologies\n\n### Core Dependencies\n\n```go\n// Networking and orchestration\ngithub.com/docker/docker // Docker API client\ngithub.com/docker/compose/v2 // Docker Compose integration\ngolang.zx2c4.com/wireguard // WireGuard implementation\ngithub.com/hashicorp/serf // Gossip protocol\n\n// State management\ngithub.com/ipfs/go-ds-crdt // CRDT distributed storage\ngithub.com/dgraph-io/badger/v3 // Embedded database\n\n// Web proxy\ngithub.com/caddyserver/caddy/v2 // HTTP server and reverse proxy\n\n// CLI and UX\ngithub.com/spf13/cobra // CLI framework\ngithub.com/charmbracelet/huh // Interactive forms\n\n// gRPC and networking\ngoogle.golang.org/grpc // gRPC framework\ngithub.com/siderolabs/grpc-proxy // gRPC proxy for forwarding\n```\n\n## Development Workflow\n\n### Build and Development\n\n```bash\n# Build binaries\ngo build -o uc ./cmd/uc\ngo build -o uncloudd ./cmd/uncloudd\n```\n\n### Key Make Targets\n\n- `proto`: Generate protobuf code\n- `ucind-cluster`: Create development cluster\n- `update-dev`: Deploy to development machines\n- `demo-reset`: Reset demo environment\n- `fmt`: Format code\n- `test`: Run all tests\n- `lint`: Lint the code using golangci-lint\n- `lint-and-fix`: Lint the code and fix issues whenever possible\n\n## CLI Commands Structure\n\nThe `uc` CLI provides these main command groups:\n\n### Machine Management\n\n```bash\nuc machine init # Initialize new cluster\nuc machine add # Add machine to cluster\nuc machine ls # List machines\nuc machine rm # Remove machine\n```\n\n### Service Management\n\n```bash\nuc run # Run container from image\nuc deploy # Deploy from compose.yaml\nuc scale # Scale service replicas\nuc ls # List services\nuc rm # Remove service\n```\n\n### Context and Connectivity\n\n```bash\nuc context ls # List available contexts\nuc context use # Switch context\n```\n\n### Global Flags\n\n- `--connect`: Connect to remote machine directly, without a config file\n- `--uncloud-config`: Override config file path\n\n## Development Guidelines\n\n### Code Organization\n\n- **Package naming**: Use clear, descriptive names\n- **Error handling**: Wrap errors with context using `fmt.Errorf`\n- **Logging**: Use structured logging with levels\n- **gRPC**: Services defined in `api/pb/`\n\n### Testing\n\n- Test files and locations\n - Unit tests alongside source files (`*_test.go`)\n - Integration tests in `test/e2e/`\n - Test fixtures in `test/fixtures/`\n- Use the `testify` library for assertions (e.g., `require.Equal`, `assert.Nil`)\n\n### Dependencies\n\n- Prefer standard library when possible\n- Pin versions in `go.mod`\n- Document rationale for external dependencies\n\n### Configuration\n\n- Support environment variables for key settings\n- Validate configuration early\n- Provide sensible defaults\n\n## Troubleshooting and Debugging\n\n### Common Issues\n\n- **Networking**: Check WireGuard status, iptables rules\n- **DNS**: Verify service discovery resolution\n- **Containers**: Use standard Docker debugging tools\n- **State sync**: Check Corrosion logs for replication issues\n\n### Debugging Tools\n\n- Standard Linux networking tools (`ping`, `traceroute`, `wireshark`)\n- Docker commands (`docker ps`, `docker logs`)\n- SSH access to machines for direct inspection\n- gRPC debugging tools\n\n### Logs and Monitoring\n\n- Systemd services (getting logs via `journalctl -u SERVICE_NAME`)\n - `uncloud` -- Uncloud daemon\n - `uncloud-corrosion` -- Corrosion process\n- Machine daemon logs\n- Container logs via Docker\n\n## File Patterns and Conventions\n\n### Important Files to Understand\n\n- `cmd/uc/main.go`: CLI entry point and command structure\n- `internal/cli/cli.go`: CLI implementation and configuration\n- `internal/machine/machine.go`: Core machine management\n- `pkg/api/`: Public API definitions\n- `misc/design.md`: Architecture and design philosophy\n- `README.md`: Repository README\n\n### Configuration Files\n\n- `go.mod/go.sum`: Go dependency management\n- `Makefile`: Build and development tasks\n- `Dockerfile`: Container build instructions forUncloud-in-Docker (used for testing)\n\nThis document should help AI assistants understand the project structure, make informed suggestions, and contribute\neffectively to the Uncloud codebase.\n\n## Documentation\n\nInstructions when generating documentation:\n\n- Use conversational language – write as if you were speaking to a friend.\n- Keep sentences simple, optimize for clarity and understanding.\n- Do not use em dashes (—) or semicolons (;) in sentences. Instead, break complex sentences into simpler ones.\n- Place the subject before the action whenever possible. Example: prefer \"The function loads data\" over \"Data is loaded\n by the function.\"\n"},"files":{"misc/design.md":"# Uncloud design\n\nUncloud aims to provide a solution for deploying and running containerized applications and services across various\nplatforms — from cloud VPSs to Raspberry Pis and powerful servers — or any combination of them.\n\nWhile existing prominent PaaS solutions like https://coolify.io/ and https://dokploy.com/ use a single machine or Docker\nSwarm as their underlying infrastructure, Uncloud takes a different approach. It aims to offer a multi-host experience\nsimilar to the serverless solutions provided by cloud providers but on users' own machines. Users should be able to add\nmachines as compute resources without worrying about control plane high availability and cluster management. For\nexample, a user could combine a cloud VPS with a spare Raspberry Pi to create a unified computing environment.\n\nUncloud aims to replace the concept of a \"cluster\" with a \"network\" of machines. It's similar to Tailscale's design\nwhere users can add machines to a network to enable secure communication between them. The primary challenges in this\ndesign are:\n\n1. Building such a network\n2. Orchestrating containers across it\n3. Exposing services to the internet\n\n## Network of machines\n\nWhile Tailscale is an obvious solution for establishing an overlay network, it has some drawbacks:\n\n1. Users must register an account and generate a key for each machine.\n2. Every container requires a Tailscale proxy to communicate with other machines and containers.\n\nThe Tailscale subnet router feature could potentially be used to route traffic from the machine to the internal Docker\nnetwork.\n\nAs an alternative, we can configure a simple flat WireGuard mesh network during the machine setup process. For example:\n\n| CIDR | Description |\n|-----------------|----------------------------------------------------------|\n| `10.210.0.0/16` | The entire WireGuard mesh network |\n| `10.210.X.0/24` | `/24` subnet assigned to machine `X` |\n| `10.210.X.1/32` | Machine `X` address is the first address from the subnet |\n| `10.210.X.Y/32` | Container `Y` address running on machine `X` |\n\nTesting has shown that inter-container and machine routing works well when `/24` is assigned to the Docker network\nbridge, while the WireGuard interface is configured with just the `/32` machine address.\n\nThis setup allows containers to communicate with each other on the same machine or across machines without address\ntranslation. This design also enables the future implementation of [ACLs](https://tailscale.com/kb/1018/acls) and\nsecurity groups to restrict traffic between machines and containers if needed.\n\nThe peer discovery and NAT traversal techniques for the WireGuard mesh are heavily inspired by the\nTalos [KubeSpan](https://www.talos.dev/v1.7/talos-guides/network/kubespan/) design.\n\n## Orchestration\n\nThe main question that drives the design of the orchestration system is can we build a system that doesn't require a\ncentralised control plane? The short answer is yes, we can. However, it's not easy as decentralised systems are\ninherently more complex than centralised ones.\n\nThe existing solutions such as Kubernetes, Docker Swarm, and Nomad use a centralised control plane to provide a single\npoint of entry for the user to interact with the cluster. I'm envisioning a system where all machines are equal, yet\neach one can represent the entire cluster and receive commands from the user. Where a user can be a CLI tool or a web\ninterface that can run on any machine in the network. In case of network partitioning, the user should continue to be\nable to interact with each partition separately to manage services running on them. The system should be able to\nreconcile the state of the network when the partition is healed. Such an architecture favors Availability and Partition\ntolerance (AP) over Consistency and Availability (CA) in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem).\n\n### Shared state\n\nOne of the ways to achieve this is to store the entire shared state on every machine and keep them all in sync. A couple\nof techniques can be used to assist with this:\na [Conflict-Free Replicated Data Type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) (CRDT) and a\npeer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol).\n\nBy composing the state using CRDT data structures allows any machine to modify its own copy of the state independently,\nconcurrently, and without coordination with other machines. The CRDT automatically resolves conflicts that might arise\nfrom concurrent updates. Although the machines may have different states at any given time, they're guaranteed to\neventually converge to the *same* state. The tradeoff is that the *same* state doesn't always mean the intended one from\neach user's point of view.\n\nThere is a distributed key-value store implementation in Go ([ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt)) from\nthe IPFS ecosystem that uses\n[Merkle CRDTs](https://research.protocol.ai/publications/merkle-crdts-merkle-dags-meet-crdts/psaras2020.pdf). It's\nindependent of the permanent storage implementation and the transport layer for broadcasting and receiving updates from\npeers.\n\nThe gossip protocol can be used to propagate state updates across the network. There is a great project from HashiCorp\ncalled [Serf](https://www.serf.io/) that provides a gossip protocol implementation in addition to cluster membership,\nfailure detection, and basic orchestration that is decentralized, fault-tolerant and highly available. It's used in\nNomad and Consul internally.\n\n[`experiment/serf_crdt.go`](../experiment/serf_crdt.go) is an experimental distributed key-value store\nusing [BadgerDB](https://github.com/dgraph-io/badger) as the persistent storage for\n[ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) and Serf as the gossip protocol that uses its custom user events\nand queries features. The implementation is very inefficient but the results are promising. A couple of machines are\nable to replicate changes to the store across the internet in a fraction of a second and converge their copies if they\nget out of sync. Although, the convergence is quite slow at the moment due to the inefficient broadcast and query\nimplementation.\n\nThere is a hypothesis that in practice there shouldn't be much churn and conflicts in the state of relatively small\ndeployments. The updates should be propagated quickly enough thus the eventually consistent behaviour should be\nacceptable.\n\n### Container scheduling\n\nThere are two fundamental approaches to design an orchestration system: declarative and imperative. I believe that\nUncloud should use a hybrid approach that favours the imperative over the declarative one where possible.\n\nThe imperative approach allows the errors to be handled in a more predictable way. While the declarative approach allows\nto decouple components of the system by using a shared state and events for communication.\n\nFor example, the user can run a command to start a container on a specific machine. The command can call the target\nmachine directly to start the container, handle the errors accordingly, and return the result to the user.\nAlternatively, the command can update the local state declaring that a container should be started on a specific machine\nand wait until it's propagated to the target machine and reconciled. Arguably, the latter approach is more complex, less\npredictable, and has more edge cases to handle.\n\nHowever, asynchronously updating the configuration of DNS servers or reverse proxies in accordance to the state changes\ncaused by started or stopped containers is likely a more reliable approach than doing it imperatively.\n\nNot sure if we really need this yet, but more complex scenarios like running and watching a replica set of containers\nmight require a sort of coordination between machines. I believe this can be achieved by using consensus and leader\nelection algorithms on demand.\n\nMaybe for now we should only aim for a simpler and more static container orchestrator where container scheduling can\nonly be initiated by a user. Docker on each machine will ensure that the containers are running and restarted on\nfailures, of course. But they won't be moved to other machines automatically.\n\n## Service discovery\n\nService discovery using DNS has its own drawbacks but it's likely the simplest way to implement it. Like in Docker and\nDocker Swarm, the Uncloud agent on every machine will expose an internal DNS server to the machine itself and the\ncontainers running on it. The DNS server will resolve machine, container, and service names to their respective IP\naddresses within the mesh network.\n\nDNS servers will watch running containers through the shared state and update their DNS records accordingly. The\nfollowing table is an example list of available DNS names:\n\n| DNS name | Resolves to |\n|----------------------------------------------------|----------------------------------------------------------------------------|\n| `.machine.internal` | Mesh IP of the machine. Machine names must be unique within the org |\n| `..machine.internal` | Mesh IP of the container on the machine |\n| `.internal` | Mesh IPs of all containers for the service |\n| `lb..internal` | Virtual IP / LB IP that balances traffic to all containers for the service |\n\nA proper thinking about the name design is needed. Most likely we need to namespace them by project or environment name\nto allow for multiple instances of the same service and avoid conflicts.\n\nVirtual IPs for services using IPVS, ip/nftables, or eBPF can be implemented later if we see fit.\n\n## Ingress\n\nServices can be exposed to the internet by running a reverse proxy on a machine(s) that listens on the public IP\naddress. A user should be able to control which machine(s) should run a reverse proxy by assigning an appropriate role\nto them.\n\nTraefik or Caddy can be used as a reverse proxy that can automatically discover services running on the network using\nthe internal DNS server. They both support automatic TLS certificate generation and renewal using Let's Encrypt.\n\nIn case of network partitioning, DNS servers should adjust their records in accordance with what containers are\navailable in their partition. Consequently, the reverse proxy should adjust its configuration to route traffic to only\nthe available containers by resolving the updated DNS records.\n","AGENTS.md":"# AGENTS.md - Uncloud Project Guide\n\nThis document provides comprehensive information about the Uncloud project for AI assistants to understand the codebase,\narchitecture, and development practices.\n\n## Project Overview\n\n**Uncloud** is a lightweight clustering and container orchestration tool that enables deployment and management of web\napplications across cloud VMs and bare metal servers. It creates a secure WireGuard mesh network between Docker hosts\nand provides automatic service discovery, load balancing, HTTPS ingress, and simple CLI commands for application\nmanagement.\n\n### Key Characteristics\n\n- **Language**: Go\n- **Architecture**: Decentralized, no control plane\n- **Target**: Self-hosted infrastructure without Kubernetes complexity\n- **License**: View LICENSE file for details\n- **Status**: Active development, not yet ready for production\n\n## Core Features\n\n### 🏗️ Infrastructure\n\n- **Multi-machine deployment**: Combine cloud VMs, dedicated servers, and bare metal\n- **Zero-config networking**: Automatic WireGuard mesh with NAT traversal\n- **Decentralized design**: No central control plane, all machines are equal\n- **Service discovery**: Built-in DNS server resolves service names to container IPs\n\n### 🚀 Application Management\n\n- **Docker Compose compatibility**: Uses familiar Docker Compose format\n- **Zero-downtime deployments**: Rolling updates without service interruption\n- **Automatic HTTPS**: Caddy reverse proxy with Let's Encrypt integration\n- **Managed DNS**: Free `*.xxxxxx.uncld.dev` subdomains via Uncloud DNS service\n- **Cross-machine scaling**: Run containers across multiple machines\n\n### 🔧 Developer Experience\n\n- **Docker-like CLI**: Familiar commands (`uc` binary)\n- **Imperative operations**: Direct commands vs. declarative state reconciliation\n- **Remote management**: Control entire infrastructure via SSH to any machine\n- **Minimal overhead**: ~150MB RAM footprint per machine\n\n## Architecture\n\n### Core Components\n\n1. **CLI (`uc`)** - Main user interface for cluster management\n2. **Daemon (`uncloudd`)** - Machine daemon running on each node\n3. **Corrosion** - Distributed SQLite database for cluster state (Fly.io project)\n4. **Caddy** - Reverse proxy for HTTPS termination and routing\n5. **WireGuard** - Secure mesh networking between machines\n\n### Network Architecture\n\n- Each machine gets unique subnet (e.g., `10.210.0.0/24`, `10.210.1.0/24`)\n- Containers get cluster-unique IPs for direct communication\n- Automatic peer discovery and key management\n- NAT traversal for machines behind firewalls\n\n### State Management\n\n- **CRDT-based distributed storage** using Corrosion\n- **Eventually consistent** state across all machines\n- **Gossip protocol** (Serf) for state propagation\n- **No quorum requirements** - partial network splits remain functional\n\n## Project Structure\n\n### Key Directories\n\n- **`cmd/`**: Contains main applications\n\n - `uc/`: CLI tool with subcommands for machine, service, volume management\n - `uncloudd/`: Daemon that runs on each machine\n - `ucind/`: Development cluster management for testing\n\n- **`internal/`**: Internal implementation packages\n\n - `cli/`: Command-line interface logic\n - `machine/`: Machine lifecycle and state management\n - `daemon/`: Daemon implementation and gRPC services\n - `dns/`: Internal DNS server for service discovery\n\n- **`pkg/`**: Public API packages for external use\n\n - `api/`: Core API types and definitions\n - `client/`: Client libraries for interacting with Uncloud\n\n- **`experiment/`**: Experimental features and prototypes\n- **`scripts/`**: Installation and utility scripts\n- **`test/`**: Test suites and test infrastructure\n- **`website/`**: Documentation website (Docusaurus)\n\n - `landing/`: Landing page\n - `docs/`: User documentation\n\n- **`misc/`**: Design documents and guides\n\n## Key Technologies\n\n### Core Dependencies\n\n```go\n// Networking and orchestration\ngithub.com/docker/docker // Docker API client\ngithub.com/docker/compose/v2 // Docker Compose integration\ngolang.zx2c4.com/wireguard // WireGuard implementation\ngithub.com/hashicorp/serf // Gossip protocol\n\n// State management\ngithub.com/ipfs/go-ds-crdt // CRDT distributed storage\ngithub.com/dgraph-io/badger/v3 // Embedded database\n\n// Web proxy\ngithub.com/caddyserver/caddy/v2 // HTTP server and reverse proxy\n\n// CLI and UX\ngithub.com/spf13/cobra // CLI framework\ngithub.com/charmbracelet/huh // Interactive forms\n\n// gRPC and networking\ngoogle.golang.org/grpc // gRPC framework\ngithub.com/siderolabs/grpc-proxy // gRPC proxy for forwarding\n```\n\n## Development Workflow\n\n### Build and Development\n\n```bash\n# Build binaries\ngo build -o uc ./cmd/uc\ngo build -o uncloudd ./cmd/uncloudd\n```\n\n### Key Make Targets\n\n- `proto`: Generate protobuf code\n- `ucind-cluster`: Create development cluster\n- `update-dev`: Deploy to development machines\n- `demo-reset`: Reset demo environment\n- `fmt`: Format code\n- `test`: Run all tests\n- `lint`: Lint the code using golangci-lint\n- `lint-and-fix`: Lint the code and fix issues whenever possible\n\n## CLI Commands Structure\n\nThe `uc` CLI provides these main command groups:\n\n### Machine Management\n\n```bash\nuc machine init # Initialize new cluster\nuc machine add # Add machine to cluster\nuc machine ls # List machines\nuc machine rm # Remove machine\n```\n\n### Service Management\n\n```bash\nuc run # Run container from image\nuc deploy # Deploy from compose.yaml\nuc scale # Scale service replicas\nuc ls # List services\nuc rm # Remove service\n```\n\n### Context and Connectivity\n\n```bash\nuc context ls # List available contexts\nuc context use # Switch context\n```\n\n### Global Flags\n\n- `--connect`: Connect to remote machine directly, without a config file\n- `--uncloud-config`: Override config file path\n\n## Development Guidelines\n\n### Code Organization\n\n- **Package naming**: Use clear, descriptive names\n- **Error handling**: Wrap errors with context using `fmt.Errorf`\n- **Logging**: Use structured logging with levels\n- **gRPC**: Services defined in `api/pb/`\n\n### Testing\n\n- Test files and locations\n - Unit tests alongside source files (`*_test.go`)\n - Integration tests in `test/e2e/`\n - Test fixtures in `test/fixtures/`\n- Use the `testify` library for assertions (e.g., `require.Equal`, `assert.Nil`)\n\n### Dependencies\n\n- Prefer standard library when possible\n- Pin versions in `go.mod`\n- Document rationale for external dependencies\n\n### Configuration\n\n- Support environment variables for key settings\n- Validate configuration early\n- Provide sensible defaults\n\n## Troubleshooting and Debugging\n\n### Common Issues\n\n- **Networking**: Check WireGuard status, iptables rules\n- **DNS**: Verify service discovery resolution\n- **Containers**: Use standard Docker debugging tools\n- **State sync**: Check Corrosion logs for replication issues\n\n### Debugging Tools\n\n- Standard Linux networking tools (`ping`, `traceroute`, `wireshark`)\n- Docker commands (`docker ps`, `docker logs`)\n- SSH access to machines for direct inspection\n- gRPC debugging tools\n\n### Logs and Monitoring\n\n- Systemd services (getting logs via `journalctl -u SERVICE_NAME`)\n - `uncloud` -- Uncloud daemon\n - `uncloud-corrosion` -- Corrosion process\n- Machine daemon logs\n- Container logs via Docker\n\n## File Patterns and Conventions\n\n### Important Files to Understand\n\n- `cmd/uc/main.go`: CLI entry point and command structure\n- `internal/cli/cli.go`: CLI implementation and configuration\n- `internal/machine/machine.go`: Core machine management\n- `pkg/api/`: Public API definitions\n- `misc/design.md`: Architecture and design philosophy\n- `README.md`: Repository README\n\n### Configuration Files\n\n- `go.mod/go.sum`: Go dependency management\n- `Makefile`: Build and development tasks\n- `Dockerfile`: Container build instructions forUncloud-in-Docker (used for testing)\n\nThis document should help AI assistants understand the project structure, make informed suggestions, and contribute\neffectively to the Uncloud codebase.\n\n## Documentation\n\nInstructions when generating documentation:\n\n- Use conversational language – write as if you were speaking to a friend.\n- Keep sentences simple, optimize for clarity and understanding.\n- Do not use em dashes (—) or semicolons (;) in sentences. Instead, break complex sentences into simpler ones.\n- Place the subject before the action whenever possible. Example: prefer \"The function loads data\" over \"Data is loaded\n by the function.\"\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/psviderski/uncloud/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md - Uncloud Project Guide\n\nThis document provides comprehensive information about the Uncloud project for AI assistants to understand the codebase,\narchitecture, and development practices.\n\n## Project Overview\n\n**Uncloud** is a lightweight clustering and container orchestration tool that enables deployment and management of web\napplications across cloud VMs and bare metal servers. It creates a secure WireGuard mesh network between Docker hosts\nand provides automatic service discovery, load balancing, HTTPS ingress, and simple CLI commands for application\nmanagement.\n\n### Key Characteristics\n\n- **Language**: Go\n- **Architecture**: Decentralized, no control plane\n- **Target**: Self-hosted infrastructure without Kubernetes complexity\n- **License**: View LICENSE file for details\n- **Status**: Active development, not yet ready for production\n\n## Core Features\n\n### 🏗️ Infrastructure\n\n- **Multi-machine deployment**: Combine cloud VMs, dedicated servers, and bare metal\n- **Zero-config networking**: Automatic WireGuard mesh with NAT traversal\n- **Decentralized design**: No central control plane, all machines are equal\n- **Service discovery**: Built-in DNS server resolves service names to container IPs\n\n### 🚀 Application Management\n\n- **Docker Compose compatibility**: Uses familiar Docker Compose format\n- **Zero-downtime deployments**: Rolling updates without service interruption\n- **Automatic HTTPS**: Caddy reverse proxy with Let's Encrypt integration\n- **Managed DNS**: Free `*.xxxxxx.uncld.dev` subdomains via Uncloud DNS service\n- **Cross-machine scaling**: Run containers across multiple machines\n\n### 🔧 Developer Experience\n\n- **Docker-like CLI**: Familiar commands (`uc` binary)\n- **Imperative operations**: Direct commands vs. declarative state reconciliation\n- **Remote management**: Control entire infrastructure via SSH to any machine\n- **Minimal overhead**: ~150MB RAM footprint per machine\n\n## Architecture\n\n### Core Components\n\n1. **CLI (`uc`)** - Main user interface for cluster management\n2. **Daemon (`uncloudd`)** - Machine daemon running on each node\n3. **Corrosion** - Distributed SQLite database for cluster state (Fly.io project)\n4. **Caddy** - Reverse proxy for HTTPS termination and routing\n5. **WireGuard** - Secure mesh networking between machines\n\n### Network Architecture\n\n- Each machine gets unique subnet (e.g., `10.210.0.0/24`, `10.210.1.0/24`)\n- Containers get cluster-unique IPs for direct communication\n- Automatic peer discovery and key management\n- NAT traversal for machines behind firewalls\n\n### State Management\n\n- **CRDT-based distributed storage** using Corrosion\n- **Eventually consistent** state across all machines\n- **Gossip protocol** (Serf) for state propagation\n- **No quorum requirements** - partial network splits remain functional\n\n## Project Structure\n\n### Key Directories\n\n- **`cmd/`**: Contains main applications\n\n - `uc/`: CLI tool with subcommands for machine, service, volume management\n - `uncloudd/`: Daemon that runs on each machine\n - `ucind/`: Development cluster management for testing\n\n- **`internal/`**: Internal implementation packages\n\n - `cli/`: Command-line interface logic\n - `machine/`: Machine lifecycle and state management\n - `daemon/`: Daemon implementation and gRPC services\n - `dns/`: Internal DNS server for service discovery\n\n- **`pkg/`**: Public API packages for external use\n\n - `api/`: Core API types and definitions\n - `client/`: Client libraries for interacting with Uncloud\n\n- **`experiment/`**: Experimental features and prototypes\n- **`scripts/`**: Installation and utility scripts\n- **`test/`**: Test suites and test infrastructure\n- **`website/`**: Documentation website (Docusaurus)\n\n - `landing/`: Landing page\n - `docs/`: User documentation\n\n- **`misc/`**: Design documents and guides\n\n## Key Technologies\n\n### Core Dependencies\n\n```go\n// Networking and orchestration\ngithub.com/docker/docker // Docker API client\ngithub.com/docker/compose/v2 // Docker Compose integration\ngolang.zx2c4.com/wireguard // WireGuard implementation\ngithub.com/hashicorp/serf // Gossip protocol\n\n// State management\ngithub.com/ipfs/go-ds-crdt // CRDT distributed storage\ngithub.com/dgraph-io/badger/v3 // Embedded database\n\n// Web proxy\ngithub.com/caddyserver/caddy/v2 // HTTP server and reverse proxy\n\n// CLI and UX\ngithub.com/spf13/cobra // CLI framework\ngithub.com/charmbracelet/huh // Interactive forms\n\n// gRPC and networking\ngoogle.golang.org/grpc // gRPC framework\ngithub.com/siderolabs/grpc-proxy // gRPC proxy for forwarding\n```\n\n## Development Workflow\n\n### Build and Development\n\n```bash\n# Build binaries\ngo build -o uc ./cmd/uc\ngo build -o uncloudd ./cmd/uncloudd\n```\n\n### Key Make Targets\n\n- `proto`: Generate protobuf code\n- `ucind-cluster`: Create development cluster\n- `update-dev`: Deploy to development machines\n- `demo-reset`: Reset demo environment\n- `fmt`: Format code\n- `test`: Run all tests\n- `lint`: Lint the code using golangci-lint\n- `lint-and-fix`: Lint the code and fix issues whenever possible\n\n## CLI Commands Structure\n\nThe `uc` CLI provides these main command groups:\n\n### Machine Management\n\n```bash\nuc machine init # Initialize new cluster\nuc machine add # Add machine to cluster\nuc machine ls # List machines\nuc machine rm # Remove machine\n```\n\n### Service Management\n\n```bash\nuc run # Run container from image\nuc deploy # Deploy from compose.yaml\nuc scale # Scale service replicas\nuc ls # List services\nuc rm # Remove service\n```\n\n### Context and Connectivity\n\n```bash\nuc context ls # List available contexts\nuc context use # Switch context\n```\n\n### Global Flags\n\n- `--connect`: Connect to remote machine directly, without a config file\n- `--uncloud-config`: Override config file path\n\n## Development Guidelines\n\n### Code Organization\n\n- **Package naming**: Use clear, descriptive names\n- **Error handling**: Wrap errors with context using `fmt.Errorf`\n- **Logging**: Use structured logging with levels\n- **gRPC**: Services defined in `api/pb/`\n\n### Testing\n\n- Test files and locations\n - Unit tests alongside source files (`*_test.go`)\n - Integration tests in `test/e2e/`\n - Test fixtures in `test/fixtures/`\n- Use the `testify` library for assertions (e.g., `require.Equal`, `assert.Nil`)\n\n### Dependencies\n\n- Prefer standard library when possible\n- Pin versions in `go.mod`\n- Document rationale for external dependencies\n\n### Configuration\n\n- Support environment variables for key settings\n- Validate configuration early\n- Provide sensible defaults\n\n## Troubleshooting and Debugging\n\n### Common Issues\n\n- **Networking**: Check WireGuard status, iptables rules\n- **DNS**: Verify service discovery resolution\n- **Containers**: Use standard Docker debugging tools\n- **State sync**: Check Corrosion logs for replication issues\n\n### Debugging Tools\n\n- Standard Linux networking tools (`ping`, `traceroute`, `wireshark`)\n- Docker commands (`docker ps`, `docker logs`)\n- SSH access to machines for direct inspection\n- gRPC debugging tools\n\n### Logs and Monitoring\n\n- Systemd services (getting logs via `journalctl -u SERVICE_NAME`)\n - `uncloud` -- Uncloud daemon\n - `uncloud-corrosion` -- Corrosion process\n- Machine daemon logs\n- Container logs via Docker\n\n## File Patterns and Conventions\n\n### Important Files to Understand\n\n- `cmd/uc/main.go`: CLI entry point and command structure\n- `internal/cli/cli.go`: CLI implementation and configuration\n- `internal/machine/machine.go`: Core machine management\n- `pkg/api/`: Public API definitions\n- `misc/design.md`: Architecture and design philosophy\n- `README.md`: Repository README\n\n### Configuration Files\n\n- `go.mod/go.sum`: Go dependency management\n- `Makefile`: Build and development tasks\n- `Dockerfile`: Container build instructions forUncloud-in-Docker (used for testing)\n\nThis document should help AI assistants understand the project structure, make informed suggestions, and contribute\neffectively to the Uncloud codebase.\n\n## Documentation\n\nInstructions when generating documentation:\n\n- Use conversational language – write as if you were speaking to a friend.\n- Keep sentences simple, optimize for clarity and understanding.\n- Do not use em dashes (—) or semicolons (;) in sentences. Instead, break complex sentences into simpler ones.\n- Place the subject before the action whenever possible. Example: prefer \"The function loads data\" over \"Data is loaded\n by the function.\"\n","isInternal":false,"tokens":1933,"sizeBytes":8670},{"name":"design.md","path":"misc/design.md","rawUrl":"https://raw.githubusercontent.com/psviderski/uncloud/HEAD/misc/design.md","title":"misc - UI Design System & Style Tokens","category":"project-spec","format":"markdown","content":"# Uncloud design\n\nUncloud aims to provide a solution for deploying and running containerized applications and services across various\nplatforms — from cloud VPSs to Raspberry Pis and powerful servers — or any combination of them.\n\nWhile existing prominent PaaS solutions like https://coolify.io/ and https://dokploy.com/ use a single machine or Docker\nSwarm as their underlying infrastructure, Uncloud takes a different approach. It aims to offer a multi-host experience\nsimilar to the serverless solutions provided by cloud providers but on users' own machines. Users should be able to add\nmachines as compute resources without worrying about control plane high availability and cluster management. For\nexample, a user could combine a cloud VPS with a spare Raspberry Pi to create a unified computing environment.\n\nUncloud aims to replace the concept of a \"cluster\" with a \"network\" of machines. It's similar to Tailscale's design\nwhere users can add machines to a network to enable secure communication between them. The primary challenges in this\ndesign are:\n\n1. Building such a network\n2. Orchestrating containers across it\n3. Exposing services to the internet\n\n## Network of machines\n\nWhile Tailscale is an obvious solution for establishing an overlay network, it has some drawbacks:\n\n1. Users must register an account and generate a key for each machine.\n2. Every container requires a Tailscale proxy to communicate with other machines and containers.\n\nThe Tailscale subnet router feature could potentially be used to route traffic from the machine to the internal Docker\nnetwork.\n\nAs an alternative, we can configure a simple flat WireGuard mesh network during the machine setup process. For example:\n\n| CIDR | Description |\n|-----------------|----------------------------------------------------------|\n| `10.210.0.0/16` | The entire WireGuard mesh network |\n| `10.210.X.0/24` | `/24` subnet assigned to machine `X` |\n| `10.210.X.1/32` | Machine `X` address is the first address from the subnet |\n| `10.210.X.Y/32` | Container `Y` address running on machine `X` |\n\nTesting has shown that inter-container and machine routing works well when `/24` is assigned to the Docker network\nbridge, while the WireGuard interface is configured with just the `/32` machine address.\n\nThis setup allows containers to communicate with each other on the same machine or across machines without address\ntranslation. This design also enables the future implementation of [ACLs](https://tailscale.com/kb/1018/acls) and\nsecurity groups to restrict traffic between machines and containers if needed.\n\nThe peer discovery and NAT traversal techniques for the WireGuard mesh are heavily inspired by the\nTalos [KubeSpan](https://www.talos.dev/v1.7/talos-guides/network/kubespan/) design.\n\n## Orchestration\n\nThe main question that drives the design of the orchestration system is can we build a system that doesn't require a\ncentralised control plane? The short answer is yes, we can. However, it's not easy as decentralised systems are\ninherently more complex than centralised ones.\n\nThe existing solutions such as Kubernetes, Docker Swarm, and Nomad use a centralised control plane to provide a single\npoint of entry for the user to interact with the cluster. I'm envisioning a system where all machines are equal, yet\neach one can represent the entire cluster and receive commands from the user. Where a user can be a CLI tool or a web\ninterface that can run on any machine in the network. In case of network partitioning, the user should continue to be\nable to interact with each partition separately to manage services running on them. The system should be able to\nreconcile the state of the network when the partition is healed. Such an architecture favors Availability and Partition\ntolerance (AP) over Consistency and Availability (CA) in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem).\n\n### Shared state\n\nOne of the ways to achieve this is to store the entire shared state on every machine and keep them all in sync. A couple\nof techniques can be used to assist with this:\na [Conflict-Free Replicated Data Type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) (CRDT) and a\npeer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol).\n\nBy composing the state using CRDT data structures allows any machine to modify its own copy of the state independently,\nconcurrently, and without coordination with other machines. The CRDT automatically resolves conflicts that might arise\nfrom concurrent updates. Although the machines may have different states at any given time, they're guaranteed to\neventually converge to the *same* state. The tradeoff is that the *same* state doesn't always mean the intended one from\neach user's point of view.\n\nThere is a distributed key-value store implementation in Go ([ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt)) from\nthe IPFS ecosystem that uses\n[Merkle CRDTs](https://research.protocol.ai/publications/merkle-crdts-merkle-dags-meet-crdts/psaras2020.pdf). It's\nindependent of the permanent storage implementation and the transport layer for broadcasting and receiving updates from\npeers.\n\nThe gossip protocol can be used to propagate state updates across the network. There is a great project from HashiCorp\ncalled [Serf](https://www.serf.io/) that provides a gossip protocol implementation in addition to cluster membership,\nfailure detection, and basic orchestration that is decentralized, fault-tolerant and highly available. It's used in\nNomad and Consul internally.\n\n[`experiment/serf_crdt.go`](../experiment/serf_crdt.go) is an experimental distributed key-value store\nusing [BadgerDB](https://github.com/dgraph-io/badger) as the persistent storage for\n[ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) and Serf as the gossip protocol that uses its custom user events\nand queries features. The implementation is very inefficient but the results are promising. A couple of machines are\nable to replicate changes to the store across the internet in a fraction of a second and converge their copies if they\nget out of sync. Although, the convergence is quite slow at the moment due to the inefficient broadcast and query\nimplementation.\n\nThere is a hypothesis that in practice there shouldn't be much churn and conflicts in the state of relatively small\ndeployments. The updates should be propagated quickly enough thus the eventually consistent behaviour should be\nacceptable.\n\n### Container scheduling\n\nThere are two fundamental approaches to design an orchestration system: declarative and imperative. I believe that\nUncloud should use a hybrid approach that favours the imperative over the declarative one where possible.\n\nThe imperative approach allows the errors to be handled in a more predictable way. While the declarative approach allows\nto decouple components of the system by using a shared state and events for communication.\n\nFor example, the user can run a command to start a container on a specific machine. The command can call the target\nmachine directly to start the container, handle the errors accordingly, and return the result to the user.\nAlternatively, the command can update the local state declaring that a container should be started on a specific machine\nand wait until it's propagated to the target machine and reconciled. Arguably, the latter approach is more complex, less\npredictable, and has more edge cases to handle.\n\nHowever, asynchronously updating the configuration of DNS servers or reverse proxies in accordance to the state changes\ncaused by started or stopped containers is likely a more reliable approach than doing it imperatively.\n\nNot sure if we really need this yet, but more complex scenarios like running and watching a replica set of containers\nmight require a sort of coordination between machines. I believe this can be achieved by using consensus and leader\nelection algorithms on demand.\n\nMaybe for now we should only aim for a simpler and more static container orchestrator where container scheduling can\nonly be initiated by a user. Docker on each machine will ensure that the containers are running and restarted on\nfailures, of course. But they won't be moved to other machines automatically.\n\n## Service discovery\n\nService discovery using DNS has its own drawbacks but it's likely the simplest way to implement it. Like in Docker and\nDocker Swarm, the Uncloud agent on every machine will expose an internal DNS server to the machine itself and the\ncontainers running on it. The DNS server will resolve machine, container, and service names to their respective IP\naddresses within the mesh network.\n\nDNS servers will watch running containers through the shared state and update their DNS records accordingly. The\nfollowing table is an example list of available DNS names:\n\n| DNS name | Resolves to |\n|----------------------------------------------------|----------------------------------------------------------------------------|\n| `.machine.internal` | Mesh IP of the machine. Machine names must be unique within the org |\n| `..machine.internal` | Mesh IP of the container on the machine |\n| `.internal` | Mesh IPs of all containers for the service |\n| `lb..internal` | Virtual IP / LB IP that balances traffic to all containers for the service |\n\nA proper thinking about the name design is needed. Most likely we need to namespace them by project or environment name\nto allow for multiple instances of the same service and avoid conflicts.\n\nVirtual IPs for services using IPVS, ip/nftables, or eBPF can be implemented later if we see fit.\n\n## Ingress\n\nServices can be exposed to the internet by running a reverse proxy on a machine(s) that listens on the public IP\naddress. A user should be able to control which machine(s) should run a reverse proxy by assigning an appropriate role\nto them.\n\nTraefik or Caddy can be used as a reverse proxy that can automatically discover services running on the network using\nthe internal DNS server. They both support automatic TLS certificate generation and renewal using Let's Encrypt.\n\nIn case of network partitioning, DNS servers should adjust their records in accordance with what containers are\navailable in their partition. Consequently, the reverse proxy should adjust its configuration to route traffic to only\nthe available containers by resolving the updated DNS records.\n","isInternal":false,"tokens":2173,"sizeBytes":10801}],"systemPromptSnippet":"\n\n# AGENTS.md - Uncloud Project Guide\n\nThis document provides comprehensive information about the Uncloud project for AI assistants to understand the codebase,\narchitecture, and development practices.\n\n## Project Overview\n\n**Uncloud** is a lightweight clustering and container orchestration tool that enables deployment and management of web\napplications across cloud VMs and bare metal servers. It creates a secure WireGuard mesh network between Docker hosts\nand provides automatic service discovery, load balancing, HTTPS ingress, and simple CLI commands for application\nmanagement.\n\n### Key Characteristics\n\n- **Language**: Go\n- **Architecture**: Decentralized, no control plane\n- **Target**: Self-hosted infrastructure without Kubernetes complexity\n- **License**: View LICENSE file for details\n- **Status**: Active development, not yet ready for production\n\n## Core Features\n\n### 🏗️ Infrastructure\n\n- **Multi-machine deployment**: Combine cloud VMs, dedicated servers, and bare metal\n- **Zero-config networking**: Automatic WireGuard mesh with NAT traversal\n- **Decentralized design**: No central control plane, all machines are equal\n- **Service discovery**: Built-in DNS server resolves service names to container IPs\n\n### 🚀 Application Management\n\n- **Docker Compose compatibility**: Uses familiar Docker Compose format\n- **Zero-downtime deployments**: Rolling updates without service interruption\n- **Automatic HTTPS**: Caddy reverse proxy with Let's Encrypt integration\n- **Managed DNS**: Free `*.xxxxxx.uncld.dev` subdomains via Uncloud DNS service\n- **Cross-machine scaling**: Run containers across multiple machines\n\n### 🔧 Developer Experience\n\n- **Docker-like CLI**: Familiar commands (`uc` binary)\n- **Imperative operations**: Direct commands vs. declarative state reconciliation\n- **Remote management**: Control entire infrastructure via SSH to any machine\n- **Minimal overhead**: ~150MB RAM footprint per machine\n\n## Architecture\n\n### Core Components\n\n1. **CLI (`uc`)** - Main user interface for cluster management\n2. **Daemon (`uncloudd`)** - Machine daemon running on each node\n3. **Corrosion** - Distributed SQLite database for cluster state (Fly.io project)\n4. **Caddy** - Reverse proxy for HTTPS termination and routing\n5. **WireGuard** - Secure mesh networking between machines\n\n### Network Architecture\n\n- Each machine gets unique subnet (e.g., `10.210.0.0/24`, `10.210.1.0/24`)\n- Containers get cluster-unique IPs for direct communication\n- Automatic peer discovery and key management\n- NAT traversal for machines behind firewalls\n\n### State Management\n\n- **CRDT-based distributed storage** using Corrosion\n- **Eventually consistent** state across all machines\n- **Gossip protocol** (Serf) for state propagation\n- **No quorum requirements** - partial network splits remain functional\n\n## Project Structure\n\n### Key Directories\n\n- **`cmd/`**: Contains main applications\n\n - `uc/`: CLI tool with subcommands for machine, service, volume management\n - `uncloudd/`: Daemon that runs on each machine\n - `ucind/`: Development cluster management for testing\n\n- **`internal/`**: Internal implementation packages\n\n - `cli/`: Command-line interface logic\n - `machine/`: Machine lifecycle and state management\n - `daemon/`: Daemon implementation and gRPC services\n - `dns/`: Internal DNS server for service discovery\n\n- **`pkg/`**: Public API packages for external use\n\n - `api/`: Core API types and definitions\n - `client/`: Client libraries for interacting with Uncloud\n\n- **`experiment/`**: Experimental features and prototypes\n- **`scripts/`**: Installation and utility scripts\n- **`test/`**: Test suites and test infrastructure\n- **`website/`**: Documentation website (Docusaurus)\n\n - `landing/`: Landing page\n - `docs/`: User documentation\n\n- **`misc/`**: Design documents and guides\n\n## Key Technologies\n\n### Core Dependencies\n\n```go\n// Networking and orchestration\ngithub.com/docker/docker // Docker API client\ngithub.com/docker/compose/v2 // Docker Compose integration\ngolang.zx2c4.com/wireguard // WireGuard implementation\ngithub.com/hashicorp/serf // Gossip protocol\n\n// State management\ngithub.com/ipfs/go-ds-crdt // CRDT distributed storage\ngithub.com/dgraph-io/badger/v3 // Embedded database\n\n// Web proxy\ngithub.com/caddyserver/caddy/v2 // HTTP server and reverse proxy\n\n// CLI and UX\ngithub.com/spf13/cobra // CLI framework\ngithub.com/charmbracelet/huh // Interactive forms\n\n// gRPC and networking\ngoogle.golang.org/grpc // gRPC framework\ngithub.com/siderolabs/grpc-proxy // gRPC proxy for forwarding\n```\n\n## Development Workflow\n\n### Build and Development\n\n```bash\n# Build binaries\ngo build -o uc ./cmd/uc\ngo build -o uncloudd ./cmd/uncloudd\n```\n\n### Key Make Targets\n\n- `proto`: Generate protobuf code\n- `ucind-cluster`: Create development cluster\n- `update-dev`: Deploy to development machines\n- `demo-reset`: Reset demo environment\n- `fmt`: Format code\n- `test`: Run all tests\n- `lint`: Lint the code using golangci-lint\n- `lint-and-fix`: Lint the code and fix issues whenever possible\n\n## CLI Commands Structure\n\nThe `uc` CLI provides these main command groups:\n\n### Machine Management\n\n```bash\nuc machine init # Initialize new cluster\nuc machine add # Add machine to cluster\nuc machine ls # List machines\nuc machine rm # Remove machine\n```\n\n### Service Management\n\n```bash\nuc run # Run container from image\nuc deploy # Deploy from compose.yaml\nuc scale # Scale service replicas\nuc ls # List services\nuc rm # Remove service\n```\n\n### Context and Connectivity\n\n```bash\nuc context ls # List available contexts\nuc context use # Switch context\n```\n\n### Global Flags\n\n- `--connect`: Connect to remote machine directly, without a config file\n- `--uncloud-config`: Override config file path\n\n## Development Guidelines\n\n### Code Organization\n\n- **Package naming**: Use clear, descriptive names\n- **Error handling**: Wrap errors with context using `fmt.Errorf`\n- **Logging**: Use structured logging with levels\n- **gRPC**: Services defined in `api/pb/`\n\n### Testing\n\n- Test files and locations\n - Unit tests alongside source files (`*_test.go`)\n - Integration tests in `test/e2e/`\n - Test fixtures in `test/fixtures/`\n- Use the `testify` library for assertions (e.g., `require.Equal`, `assert.Nil`)\n\n### Dependencies\n\n- Prefer standard library when possible\n- Pin versions in `go.mod`\n- Document rationale for external dependencies\n\n### Configuration\n\n- Support environment variables for key settings\n- Validate configuration early\n- Provide sensible defaults\n\n## Troubleshooting and Debugging\n\n### Common Issues\n\n- **Networking**: Check WireGuard status, iptables rules\n- **DNS**: Verify service discovery resolution\n- **Containers**: Use standard Docker debugging tools\n- **State sync**: Check Corrosion logs for replication issues\n\n### Debugging Tools\n\n- Standard Linux networking tools (`ping`, `traceroute`, `wireshark`)\n- Docker commands (`docker ps`, `docker logs`)\n- SSH access to machines for direct inspection\n- gRPC debugging tools\n\n### Logs and Monitoring\n\n- Systemd services (getting logs via `journalctl -u SERVICE_NAME`)\n - `uncloud` -- Uncloud daemon\n - `uncloud-corrosion` -- Corrosion process\n- Machine daemon logs\n- Container logs via Docker\n\n## File Patterns and Conventions\n\n### Important Files to Understand\n\n- `cmd/uc/main.go`: CLI entry point and command structure\n- `internal/cli/cli.go`: CLI implementation and configuration\n- `internal/machine/machine.go`: Core machine management\n- `pkg/api/`: Public API definitions\n- `misc/design.md`: Architecture and design philosophy\n- `README.md`: Repository README\n\n### Configuration Files\n\n- `go.mod/go.sum`: Go dependency management\n- `Makefile`: Build and development tasks\n- `Dockerfile`: Container build instructions forUncloud-in-Docker (used for testing)\n\nThis document should help AI assistants understand the project structure, make informed suggestions, and contribute\neffectively to the Uncloud codebase.\n\n## Documentation\n\nInstructions when generating documentation:\n\n- Use conversational language – write as if you were speaking to a friend.\n- Keep sentences simple, optimize for clarity and understanding.\n- Do not use em dashes (—) or semicolons (;) in sentences. Instead, break complex sentences into simpler ones.\n- Place the subject before the action whenever possible. Example: prefer \"The function loads data\" over \"Data is loaded\n by the function.\"\n\n\n# Uncloud design\n\nUncloud aims to provide a solution for deploying and running containerized applications and services across various\nplatforms — from cloud VPSs to Raspberry Pis and powerful servers — or any combination of them.\n\nWhile existing prominent PaaS solutions like https://coolify.io/ and https://dokploy.com/ use a single machine or Docker\nSwarm as their underlying infrastructure, Uncloud takes a different approach. It aims to offer a multi-host experience\nsimilar to the serverless solutions provided by cloud providers but on users' own machines. Users should be able to add\nmachines as compute resources without worrying about control plane high availability and cluster management. For\nexample, a user could combine a cloud VPS with a spare Raspberry Pi to create a unified computing environment.\n\nUncloud aims to replace the concept of a \"cluster\" with a \"network\" of machines. It's similar to Tailscale's design\nwhere users can add machines to a network to enable secure communication between them. The primary challenges in this\ndesign are:\n\n1. Building such a network\n2. Orchestrating containers across it\n3. Exposing services to the internet\n\n## Network of machines\n\nWhile Tailscale is an obvious solution for establishing an overlay network, it has some drawbacks:\n\n1. Users must register an account and generate a key for each machine.\n2. Every container requires a Tailscale proxy to communicate with other machines and containers.\n\nThe Tailscale subnet router feature could potentially be used to route traffic from the machine to the internal Docker\nnetwork.\n\nAs an alternative, we can configure a simple flat WireGuard mesh network during the machine setup process. For example:\n\n| CIDR | Description |\n|-----------------|----------------------------------------------------------|\n| `10.210.0.0/16` | The entire WireGuard mesh network |\n| `10.210.X.0/24` | `/24` subnet assigned to machine `X` |\n| `10.210.X.1/32` | Machine `X` address is the first address from the subnet |\n| `10.210.X.Y/32` | Container `Y` address running on machine `X` |\n\nTesting has shown that inter-container and machine routing works well when `/24` is assigned to the Docker network\nbridge, while the WireGuard interface is configured with just the `/32` machine address.\n\nThis setup allows containers to communicate with each other on the same machine or across machines without address\ntranslation. This design also enables the future implementation of [ACLs](https://tailscale.com/kb/1018/acls) and\nsecurity groups to restrict traffic between machines and containers if needed.\n\nThe peer discovery and NAT traversal techniques for the WireGuard mesh are heavily inspired by the\nTalos [KubeSpan](https://www.talos.dev/v1.7/talos-guides/network/kubespan/) design.\n\n## Orchestration\n\nThe main question that drives the design of the orchestration system is can we build a system that doesn't require a\ncentralised control plane? The short answer is yes, we can. However, it's not easy as decentralised systems are\ninherently more complex than centralised ones.\n\nThe existing solutions such as Kubernetes, Docker Swarm, and Nomad use a centralised control plane to provide a single\npoint of entry for the user to interact with the cluster. I'm envisioning a system where all machines are equal, yet\neach one can represent the entire cluster and receive commands from the user. Where a user can be a CLI tool or a web\ninterface that can run on any machine in the network. In case of network partitioning, the user should continue to be\nable to interact with each partition separately to manage services running on them. The system should be able to\nreconcile the state of the network when the partition is healed. Such an architecture favors Availability and Partition\ntolerance (AP) over Consistency and Availability (CA) in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem).\n\n### Shared state\n\nOne of the ways to achieve this is to store the entire shared state on every machine and keep them all in sync. A couple\nof techniques can be used to assist with this:\na [Conflict-Free Replicated Data Type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) (CRDT) and a\npeer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol).\n\nBy composing the state using CRDT data structures allows any machine to modify its own copy of the state independently,\nconcurrently, and without coordination with other machines. The CRDT automatically resolves conflicts that might arise\nfrom concurrent updates. Although the machines may have different states at any given time, they're guaranteed to\neventually converge to the *same* state. The tradeoff is that the *same* state doesn't always mean the intended one from\neach user's point of view.\n\nThere is a distributed key-value store implementation in Go ([ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt)) from\nthe IPFS ecosystem that uses\n[Merkle CRDTs](https://research.protocol.ai/publications/merkle-crdts-merkle-dags-meet-crdts/psaras2020.pdf). It's\nindependent of the permanent storage implementation and the transport layer for broadcasting and receiving updates from\npeers.\n\nThe gossip protocol can be used to propagate state updates across the network. There is a great project from HashiCorp\ncalled [Serf](https://www.serf.io/) that provides a gossip protocol implementation in addition to cluster membership,\nfailure detection, and basic orchestration that is decentralized, fault-tolerant and highly available. It's used in\nNomad and Consul internally.\n\n[`experiment/serf_crdt.go`](../experiment/serf_crdt.go) is an experimental distributed key-value store\nusing [BadgerDB](https://github.com/dgraph-io/badger) as the persistent storage for\n[ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) and Serf as the gossip protocol that uses its custom user events\nand queries features. The implementation is very inefficient but the results are promising. A couple of machines are\nable to replicate changes to the store across the internet in a fraction of a second and converge their copies if they\nget out of sync. Although, the convergence is quite slow at the moment due to the inefficient broadcast and query\nimplementation.\n\nThere is a hypothesis that in practice there shouldn't be much churn and conflicts in the state of relatively small\ndeployments. The updates should be propagated quickly enough thus the eventually consistent behaviour should be\nacceptable.\n\n### Container scheduling\n\nThere are two fundamental approaches to design an orchestration system: declarative and imperative. I believe that\nUncloud should use a hybrid approach that favours the imperative over the declarative one where possible.\n\nThe imperative approach allows the errors to be handled in a more predictable way. While the declarative approach allows\nto decouple components of the system by using a shared state and events for communication.\n\nFor example, the user can run a command to start a container on a specific machine. The command can call the target\nmachine directly to start the container, handle the errors accordingly, and return the result to the user.\nAlternatively, the command can update the local state declaring that a container should be started on a specific machine\nand wait until it's propagated to the target machine and reconciled. Arguably, the latter approach is more complex, less\npredictable, and has more edge cases to handle.\n\nHowever, asynchronously updating the configuration of DNS servers or reverse proxies in accordance to the state changes\ncaused by started or stopped containers is likely a more reliable approach than doing it imperatively.\n\nNot sure if we really need this yet, but more complex scenarios like running and watching a replica set of containers\nmight require a sort of coordination between machines. I believe this can be achieved by using consensus and leader\nelection algorithms on demand.\n\nMaybe for now we should only aim for a simpler and more static container orchestrator where container scheduling can\nonly be initiated by a user. Docker on each machine will ensure that the containers are running and restarted on\nfailures, of course. But they won't be moved to other machines automatically.\n\n## Service discovery\n\nService discovery using DNS has its own drawbacks but it's likely the simplest way to implement it. Like in Docker and\nDocker Swarm, the Uncloud agent on every machine will expose an internal DNS server to the machine itself and the\ncontainers running on it. The DNS server will resolve machine, container, and service names to their respective IP\naddresses within the mesh network.\n\nDNS servers will watch running containers through the shared state and update their DNS records accordingly. The\nfollowing table is an example list of available DNS names:\n\n| DNS name | Resolves to |\n|----------------------------------------------------|----------------------------------------------------------------------------|\n| `.machine.internal` | Mesh IP of the machine. Machine names must be unique within the org |\n| `..machine.internal` | Mesh IP of the container on the machine |\n| `.internal` | Mesh IPs of all containers for the service |\n| `lb..internal` | Virtual IP / LB IP that balances traffic to all containers for the service |\n\nA proper thinking about the name design is needed. Most likely we need to namespace them by project or environment name\nto allow for multiple instances of the same service and avoid conflicts.\n\nVirtual IPs for services using IPVS, ip/nftables, or eBPF can be implemented later if we see fit.\n\n## Ingress\n\nServices can be exposed to the internet by running a reverse proxy on a machine(s) that listens on the public IP\naddress. A user should be able to control which machine(s) should run a reverse proxy by assigning an appropriate role\nto them.\n\nTraefik or Caddy can be used as a reverse proxy that can automatically discover services running on the network using\nthe internal DNS server. They both support automatic TLS certificate generation and renewal using Let's Encrypt.\n\nIn case of network partitioning, DNS servers should adjust their records in accordance with what containers are\navailable in their partition. Consequently, the reverse proxy should adjust its configuration to route traffic to only\nthe available containers by resolving the updated DNS records.\n"}