File: README.md
Tart is a virtualization toolset to build, run and manage macOS and Linux virtual machines (VMs) on Apple Silicon.
Built by CI engineers for your automation needs. Here are some highlights of Tart:
- Tart uses Apple's own
Virtualization.Frameworkfor near-native performance. - Push/Pull virtual machines from any OCI-compatible container registry.
- Use Tart Packer Plugin to automate VM creation.
- Easily integrates with any CI system.
Many companies are using Tart in their internal setups. Here are just a few of them:
Note: If your company or project is using Tart please consider sharing with the community.
Usage
Try running a Tart VM on your Apple Silicon device running macOS 13.0 (Ventura) or later (will download a 25 GB image):
brew install openai/tools/tart
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest tahoe-base
tart run tahoe-basePlease check the official documentation for more information and/or feel free to use discussions
for remaining questions.
File: benchmark/README.md
Benchmark
Tart comes with a Golang-based benchmarking utility that allows one to easily compare host and guest performance.
Currently, only Flexible I/O tester workloads are supported. To run them, first make sure that passwordless sudo is configured.
Then, install Golang. The easiest way is through Homebrew:
brew install goFinally, run the following command from this (benchmark/) directory:
go run cmd/main.go fio --image ghcr.io/cirruslabs/macos-tahoe-base:latest --prepare 'sudo purge && sync'You can also enable the debugging output to diagnose issues:
go run cmd/main.go fio --debugTo compare an empty Tart home with the same pull after its immutable base has
been prewarmed, provide a standalone remote base image and a stacked image built
from it. For example, create and push a stacked child of the public Tahoe base:
BASE_IMAGE=ghcr.io/cirruslabs/macos-tahoe-base:latest
STACKED_IMAGE=ghcr.io/your-org/macos-tahoe-stacked:latest
tart clone --stacked "$BASE_IMAGE" macos-tahoe-stacked
tart push macos-tahoe-stacked "$STACKED_IMAGE"Then benchmark that pair:
go run cmd/main.go stacked-oci \
--base-image "$BASE_IMAGE" \
--image "$STACKED_IMAGE"The command first performs an unmeasured pull to warm registry, CDN, and
filesystem caches. It then uses disposable TART_HOME directories for both
measured scenarios. The prewarmed scenario keeps the VM created bytart clone --stacked alive while pulling the child, so the shared immutable
base layer remains referenced and available for reuse.
Results
Mar 27, 2024
Host:
- Hardware: Mac mini (Apple M2 Pro, 8 performance and 4 efficiency cores, 32 GB RAM,
Mac14,12) - OS: macOS Sonoma 14.4.1
Guest:
- Hardware: Virtualization.Framework
- OS: macOS Sonoma 14.4.1
Name Executor Bandwidth I/O operations
Random writing of 1MB local 2.6 GB/s 649.35 kIOPS
Random writing of 1MB Tart 2.5 GB/s 620.22 kIOPS
Random writing of 10MB local 2.6 GB/s 651.74 kIOPS
Random writing of 10MB Tart 2.5 GB/s 615.52 kIOPS
Random writing of 100MB local 1.9 GB/s 481.51 kIOPS
Random writing of 100MB Tart 2.0 GB/s 493.31 kIOPS
Random writing of 1000MB local 1.7 GB/s 414.89 kIOPS
Random writing of 1000MB Tart 1.1 GB/s 287.4 kIOPSDec 2, 2024
Host:
- Hardware: MacBook Pro (Apple M1 Pro, 8 performance and 2 efficiency cores, 32 GB RAM,
MacBookPro18,3) - OS: macOS Sequoia 15.1.1
Guest:
- Hardware: Virtualization.Framework
- OS: macOS Sonoma 14.6
/* Detailed source-code truncated for AI context efficiency. */Dec 4, 2024
Host:
- AWS instance:
mac2.metal+gp3EBS volume - Hardware: Mac mini (Apple M1, 4 performance and 4 efficiency cores, 16 GB RAM,
Macmini9,1) - OS: macOS Sequoia 15.0
Guest:
- Hardware: Virtualization.Framework
- OS: macOS Sonoma 14.6
/* Detailed source-code truncated for AI context efficiency. */Host:
- AWS instance:
mac2.metal+gp3EBS volume - Hardware: Mac mini (Apple M1, 4 performance and 4 efficiency cores, 16 GB RAM,
Macmini9,1) - OS: macOS Sequoia 15.0
Guest:
- Hardware: Virtualization.Framework
- OS: macOS Sequoia 15.1
/* Detailed source-code truncated for AI context efficiency. */March 23, 2025
Host:
- Hardware: Mac mini (Apple M2 Pro, 8 performance and 4 efficiency cores, 32 GB RAM,
Mac14,12) - OS: macOS Sequoia 15.3.2
- Xcode: 16.2
Guest:
- Hardware: Virtualization.Framework
- OS: macOS Sonoma 15.3.2
- Xcode: 16.2
Name Executor Time
XcodeBenchmark (d869315) local 2m19s
XcodeBenchmark (d869315) Tart 3m59s
XcodeBenchmark (d869315) Tart (--root-disk-opts="sync=none") 3m48s
XcodeBenchmark (d869315) Tart (--root-disk-opts="caching=cached") 3m35s
XcodeBenchmark (d869315) Tart (--root-disk-opts="sync=none,caching=cached") 3m14sFile: docs/blog/posts/2023-02-11-changing-tart-license.md
draft: false
date: 2023-02-11
search:
exclude: true
authors:
- fkorotkov
categories: - announcement
Changing Tart License
!!! note "Current license"
This post describes a historical license change announced on February 11, 2023.
As of June 5, 2026, Tart is maintained by OpenAI and licensed under
FSL-1.1-ALv2.
The usage limits, paid tiers, pricing, support commitments, and contact details described below no longer apply.
TLDR: We are transitioning Tart's licensing from AGPL-3.0 to Fair Source 100. This change will
permit unlimited installations on personal computers, but organizations that exceed a certain number of server
installations utilizing 100 CPU cores will be required to obtain a paid license.
Background
Exactly a year ago on February 11th 2022 we started working on Tart β a tiny CLI to run macOS virtual machines on Apple Silicon.
Three months later we successfully started using Tart in our own production system and decided to share Tart with everyone.
The goal was to establish a community of users and contributors to transform Tart from a small CLI to a robust tool
for various scenarios. Unfortunately, we were not successful in attracting a significant number of contributors.
It's important to note that we did have seven individuals who contributed to the development of Tart to the best of
their abilities. However, one of the challenges of contributing to Tart is that the skill set required for a contribution
is vastly different from the skill set typically possessed by regular Tart users in their daily work. Specifically,
a contributor needs to have knowledge of the Swift programming language, as well as a background in operating systems
and network stack. This is the reason why 98.8% of the code and all the major features were contributed by Cirrus Labs engineers.
Tart is experiencing significant success among users and has seen widespread adoption for various applications.
The latest macOS Ventura virtual machine image has been downloaded over 27,000 times! We are continually receiving
feedback from an increasing number of users who are utilizing Tart in ways we had not initially anticipated. However,
with a growing user base comes a rise in requests for new features and enhancements. It can be challenging to justify
dedicating our engineering resources to meeting these demands when they do not align with the needs of our company, Cirrus Labs.
As a small, self-funded organization, our priority is to provide for our employees and their families along with developing great products.
In addition, the decision to use AGPL-3.0 as the license for Tart was not thoroughly considered at the time of its release.
The choice was made because many companies that were commercializing their products had recently switched to the AGPL license.
However, AGPL has a reputation for being viral, open to interpretation, and not in line with current standards. Additionally,
many organizations have policies against using any AGPL-licensed software in their stacks, which has limited Tart's potential
for wider adoption. See Google's AGPL policy, for example.
In order to ensure Tart's long-term viability and to allow us to allocate engineering resources towards further improving Tart,
we plan to transition to a licensing model that includes a nominal fee for companies that reach a substantial level of usage.
What is changing
In the near future, we are set to launch the first version of Orchard for Tart, a tool that facilitates the coordination
of Tart virtual machines on a cluster of Apple Silicon servers. Concurrently, we will also release version 1.0.0 of Tart,
which will establish a stable API and offer long-term support under a new Fair Source 100 license.
The Fair Source 100 license for Tart means that once a certain threshold of server installations utilizing 100 CPU cores
is exceeded, a paid license will be required. A "server installation" refers to the installation of Tart on a physical
device without a physical display connected. For example, a Mac Mini with a HDMI Dummy Plug is considered a server,
but a Mac Mini on a desk with a connected physical display is considered a personal computer. Usage on personal computers
and before reaching the 100 CPU cores limit is royalty-free and does not have the viral properties of AGPL.
When an organization surpasses the 100 CPU cores limit, they will be required to obtain a Gold Tier License,
which costs $12,000 per year. Upon reaching a limit of 500 CPU cores, a Platinum Tier License
($36,000 per year) will be required, and for organizations that exceed 3000 CPU cores, a custom Diamond Tier License
($12 per core per year) will be necessary. All paid license tiers will include priority feature development and SLAs on support with urgent issues.
Have we considered alternatives?
We have evaluated other options. Initially, we reached out to some of our largest users and asked them to consider
sponsoring the development of features that they were interested in. However, we received no response or were eventually
ignored. Another option we considered was using the open core model and developing enterprise-specific features. However,
this approach is not addressing concerns related to the viral nature of AGPL for non-enterprise users. Ultimately,
we concluded that transitioning to a source-available model with a mandatory paid licensing is fair, as the licensing fees
are relatively insignificant for companies that reach a significant level of usage.
If the new licensing model is not suitable for your organization, you are welcome to continue using the AGPL version of Tart,
but please ensure it is not used in a non-AGPL environment.
File: docs/blog/posts/2023-04-25-orchard-ga.md
draft: false
date: 2023-04-25
search:
exclude: true
authors:
- fkorotkov
categories: - announcement
- orchard
Announcing Orchard orchestration for managing macOS virtual machines at scale
Today we are happy to announce general availability of Orchard β our new orchestrator to manage Tart virtual machines at scale.
In this post weβll cover the motivation behind creating yet another orchestrator and why we didnβt go with Kubernetes or Nomad integration.
What problem are we trying to solve?
After releasing Tart we pretty quickly started getting requests about managing macOS virtual machines on a cluster of
Apple Silicon machines rather than just a single host which only allows a maximum of two virtual machines at a time.
By the end of 2022 the requests reached a tipping point, and we started planning.
First, we established some constraints about the end users and potential workload our solution should handle.
Running macOS or Linux virtual machines on Apple Silicon is a very niche use case. These VMs are either used in
automation solutions like CI/CD or for managing remote desktop environments. In this case we are aiming to manage
only thousands of virtual machines and not millions.
Second, operators of such solutions wonβt have experience of operating Kubernetes or Nomad. Operators will most likely
come with experience of using such systems but not managing them. And again, having built-in things like RBAC and
ability to scale to millions were appealing but it seemed like it would be a solution for a few rather than a solution
for everybody to use. Additionally Orchard should provide first class support for accessing virtual machines over SSH/VNC
and support script execution.
By that time, the idea of building a simple opinionated orchestrator got more and more appealing. Plus we kind of already did it
for Cirrus CIβs persistent workers feature.
Technical constraints
With the UX constraints and expectations in place we started thinking about architecture for the orchestrator that we
started calling Orchard.
Since Orchard will manage a maximum of a couple thousands virtual machines and not millions we decided to not think much
about horizontal scalability. Just a single instance of Orchard controller should be enough if it can restart quickly and
persist state between restarts.
Orchard should be secure by default. All the communication between a controller and workers should be secure.
All external API requests to Orchard controller should be authorized.
During development itβs crucial to have a quick feedback cycle. It should be extremely easy to run Orchard in development.
Configuring a production cluster should be also easy for novice operators.
High-level implementation details
Cirrus Labs started as a predominantly Kotlin shop with a little Go. But over the years we gradually moved a lot of things to Go.
We love the expressibility of Kotlin as a language but the ecosystem for writing system utilities and services is superb in Go.
Orchard is a single Go project that implements both controller server interface and worker client logic in a single repository.
This simplifies code sharing and testability of the both components and allows to change them in a single pull request.
Another benefit is that Orchard can be distributed as a single binary. We intend to run Orchard controller on a single host.
Data model for the orchestration didnβt look complex as well. These observations lead us to exploring the use of an embedded database.
Just imagine! Orchard can be distributed as a single binary with no external dependencies on any database or runtime!
And we did exactly that! Orchard is distributed as a single binary that can be run in βcontrollerβ mode on a Linux/macOS host and
in βworkerβ mode on macOS hosts. Orchard controller is using extremely fast BadgerDB key-value storage to persist data.
Conclusion
Please give Orchard a try! To run it locally in development mode on any Apple Silicon device
please run the following command:
brew install cirruslabs/cli/orchard
orchard devThis will launch a development cluster with a single worker on your machine. Refer to Orchard documentation
on how to create your first virtual machine and access it.
In a separate blog post
weβll cover how Orchard implements seamless SSH access over a gRPC connection. Stay tuned and please donβt hesitate to
open an issue!
File: docs/blog/posts/2023-04-28-orchard-ssh-over-grpc.md
draft: false
date: 2023-04-28
search:
exclude: true
authors:
- edigaryev
categories: - orchard
SSH over gRPC or how Orchard simplifies accessing VMs in private networks
We started developing Orchard, an orchestrator for Tart, with the requirement that it should allow users to access virtual machines running on worker nodes in private networks that users might not have access to.
At the same time, we wanted to enable users to access VMs on these remote workers just as easily as theyβd access network services on their local Tart VMs.
While these features sound great on paper, they pose a technical problem: how do we connect to the remote workers, let alone VMs running on these workers, if we canβt assume that these workers will be easily reachable? And how do we establish an SSH connection with a VM running on a remote worker through all these hoops?
Implementing port forwarding: gRPC to the rescue
We need to keep a full-duplex connection with the controller for the port-forwarding to work, and the two obvious protocol options are:
- WebSocket API through a new controllerβs REST API endpoint
- gRPC using
Content-Typedifferentiation
Weβve chosen the gRPC for controller βοΈ worker connection, simply because it requires less code on our side and it will only be used internally, which means we donβt need to document it as extensively as our REST API. In essence, port forwarding is streaming of bytes of a connection in both ways, so gRPC streams looked like a natural solution. The resulting protocol is dead simple:
service Controller {
rpc Watch(google.protobuf.Empty) returns (stream WatchInstruction);
rpc PortForward(stream PortForwardData) returns (stream PortForwardData);
}
message WatchInstruction {
message PortForward {
string session = 1;
string vm_uid = 2;
uint32 vm_port = 3;
}
oneof action {
PortForward port_forward_action = 1;
}
}
message PortForwardData {
bytes data = 1;
}On bootstrap, each Orchard worker establishes a Watch() RPC stream and waits for the PortForward instruction from the controller indefinitely. This long-running session might be used not just for port-forwarding, but for notifying the workers about changed resources, which results in workers picking up your VM for execution instantly.
Once PortForward instruction is received, the worker connects to the specified VM and port locally and opens a new PortForward() RPC stream with the controller, carrying the unique session identifier in the gRPC metadata to help distinguish several port forwarding requests.
Weβre using a pretty ingenious Golang package that turns any gRPC stream into a net.Conn. This allows us to abstract from the gRPC details and simply proxy two net.Conns, thus providing the port forwarding functionality.
Weβve also initially considered using Yamux to only keep a single connection with each worker, however, that involves the burden of dealing with flow control and potential implementation bugs associated with it, so weβve decided to simply open an additional connection for each port forwarding session and let the OS deal with it.
Building on top of the port-forwarding
First of all, weβve made the new port-forwarding functionality available for integrations via the Orchardβs REST API:
All you need is to use a WebSocket client when accessing this endpoint to make it work.
Secondly, weβve exposed three commands in the Orchard CLI that all use this endpoint:
`orchard port-forward`
Opens a TCP port locally and forwards everything sent to it to the specified VM (and vice versa).
For example, orchard port-forward vm sonoma-builder 2222:22 will forward traffic from the local TCP port 2222 to the ventura-builder VMβs TCP port 22.
`orchard ssh`
Connects to the specified VM on the default SSH port 22, optionally only launching a command (if specified), similarly to what the official OpenSSH client does.
For example, orchard ssh vm sonoma-builder will open an interactive session with the ventura-builder VM.
You can also send local scripts for execution by utilizing redirection:
orchard ssh vm sonoma-builder 'sh -s' < script.sh`orchard vnc`
Establishes a port forwarding to the specified VMβs default VNC port 5900 and opens the default macOS Screen Sharing app.
For example, orchard vnc vm sonoma-builder will establish a port-forwarding to the ventura-builder VM's port 5900 under the hood and launch macOS Screen Sharing app.
Note that the SSH and VNC commands expect the VM resource to specify credentials in itβs definition (can be done via orchard create vm), and will otherwise fall back to the credentials specified by --username and --password, or if none specified β to de-facto standard of admin:admin credentials.
Conclusion
Overall, the technology described in this article somewhat resembles what we previously did for Cirrus Terminal. The only difference is that in Cirrus Terminal we carry terminal-specific characters, and in Orchard β we carry bytes for an arbitrary TCP connection.
We really hope this feature will be useful for many, just as the Cirrus Terminal, and that it will remove the pain of scaling Tart beyond a single machine.
You can give Orchard a try by running it locally in development mode on any Apple Silicon device:
brew install cirruslabs/cli/orchard
orchard devThis will launch a development cluster with a single worker on your machine. Refer to Orchard documentation
on how to create your first virtual machine and access it.
Stay tuned and donβt hesitate to send us your feedback either on GitHubΒ orΒ Twitter!
File: docs/blog/posts/2023-09-20-tart-2.0.0.md
draft: false
date: 2023-09-20
search:
exclude: true
authors:
- fkorotkov
categories: - announcement
Tart 2.0.0 and community updates
Today we'd like to share some news and updates around the Tart ecosystem since the Tart 1.0.0 release back in February.
Community Growth
In the last 7 months Tart community almost tripled and growth is continuing to accelerate. Tart just crossed 25,000 installations,
dozens of companies that we know of are using Tart in their daily workflows. If your company is not in the list please consider
joining!
- { height="65" }
- { height="65" }
- { height="65" }
- { height="65" }
- { height="65" }
- { height="65" }
- { height="65" }
- { height="65" }
Recent updates and what's changing in Tart 2.0.0
In the last 7 months we've had 12 feature releases that brought a lot of features requested by the community. Here are just
a few of them to highlight:
-Custom GitLab Runner Executor.
-Cluster Management via Orchard.
-Numerous compatibility improvements for all kinds of OCI-registries.
-Sonoma Support (see details below).
But one of the most requested features/complaints was around pulling huge Tart images from remote OCI-compatible registries.
With an ideal network conditions tart pull worked pretty good but in case of any network issues it was required to
restart the pull from scratch. Additionally, some registries are notably slow streaming a single blob but can stream
multiple blobs in parallel. Finally, the initial format of storing Tart VMs was very naive: disk image is compressed
via a single stream which is chunked up into blobs that are serially uploaded to a registry. A single compression stream
means that Tart can also only decompress blobs serially.
Given these three observations above we came up with an improved format of storing Tart VM disk images. In Tart 2.0.0
disk images are chunked up first and compressed independently into blobs, when pushed, each blob has attached annotations
of expected uncompressed size and a checksum. This way when Tart 2.0.0 is pulling an image pushed by Tart 2.0.0 each blob can
be pulled, uncompressed and written at the right offset independently. Having checksums along expected uncompressed blob size
also allowed to support resumable pulls. Upon a failure Tart 2.0.0 will compare checksums of chunks and will continue pulling
only missing blobs.
Overall in our experiments we saw a 10% improvement in compressed size of the images and 4 times faster pulls.
In order to try the new image format please upgrade Tart and try to pull any of the Sonoma images:
brew upgrade cirruslabs/cli/tart
tart pull ghcr.io/cirruslabs/macos-sonoma-base:latestmacOS Sonoma Updates
Tart VMs now can be run in a "suspendable" mode which will enable VM snapshotting instead of the standard shutdown.
VMs with an existing snapshot will run from the same state as they got snapshotted. Please check demo down below:
Tart 1.8.0 brings macOS Sonoma updates! π Now you can suspend and resume your virtual machines for even faster startup times. Check out the demo below π [pic.twitter.com/RoRFT8Nwst](https://t.co/RoRFT8Nwst)
— Cirrus Labs (@cirrus_labs) [July 7, 2023](https://twitter.com/cirrus_labs/status/1677308360385765382?ref_src=twsrc%5Etfw)
<script src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>There are two caveats to the "suspendable" mode support:
- Both host and guest should be running macOS Sonoma.
- Snapshots are locally encrypted and can't be shared between physical hosts. Therefore
tart pushwon't push the corresponding snapshotted state of the VM.
Try the "suspendable" mode for yourself by passing --suspendable flag to a tart run command:
tart clone ghcr.io/cirruslabs/macos-sonoma-base:latest sonoma-base
tart run --suspendable sonoma-baseConclusion
We are very excited about this major release of Tart. Please give it a try and let us know how it went!
Stay tuned for new updates and announcements! There are a few coming up very shortly...
File: docs/blog/posts/2023-11-03-cirrus-runners-dashboard.md
draft: false
date: 2023-11-03
search:
exclude: true
authors:
- fkorotkov
categories: - announcement
New dashboard with insights into performance of Cirrus Runners
This month we are celebrating one year since launching Cirrus Runners β managed Apple Silicon infrastructure for your
GitHub Actions. During the last 12 months we ran millions of workflows for our customers and now ready to share some insights
into price performance of them for our customers.
One of the key difference with Cirrus Runners is how they are getting billed for. Customers purchase Cirrus Runners via monthly subscription
that costs $150 per each Cirrus Runner. Each runner can be used 24 hours a day 7 days a week to run GitHub Actions workflows
for an organization. If there are more outstanding jobs than available runners then they are queued and executed as soon as
there is a free runner. This is different from how GitHub-managed GitHub Actions are billed for β you pay for each minute of execution time.
The benefit of a fixed price is that you can run as many jobs as you want without worrying about the cost. The downside is that
you need to make sure that you are using your runners efficiently. This is where the new dashboard comes in handy.
But first, let's see theoretically the lowest price per minute of a Cirrus Runners. If you run 24 hours a day 7 days a week
then you will get 43,200 minutes of execution time per month. This means that the price per minute is $0.0035 if your runners
utilization is 100%. But even if your engineering teams is located in a single time zone and works 8 hours a day 5 days a week
then you will get 9,600 minutes of execution time per month which comes down to $0.015 per-minute. This is still more than 10 times cheaper
than recently announced Apple Silicon GitHub-manged runners that cost $0.16 per minute.
Now lets take a look at the new Cirrus Runners dashboard of a real customers that run their workflows on Cirrus Runners
and practically pushing the price performance pretty close to the theoretical minimum.
As you can see above Cirrus Runners Dashboard focuses on 4 core metrics:
- Minutes Used β overall amount of minutes that Cirrus Runners were executing jobs.
- Workflow Runs β absolute number of workflow runs that were executed on Cirrus Runners.
- Queue Size β number of jobs that were queued and waiting for a free Cirrus Runner.
- Queue Time β average time that jobs were waiting in the queue.
In this particular example price performance of Cirrus Runners is $0.006 per minute which is 2 times more than the theoretical minimum
and 26 times better than GitHub-managed Apple Silicon runners. But this is a extreme example, looking at queue time and queue size
we can see that the downside of such great price performance is that jobs are waiting in the queue on average around 5 minutes.
Here is another example of Cirrus Runners Dashboard for a different customer that has a slightly higher price performance of $0.017 per minute
but at the same time doesn't experience queue time at all. Note that $0.017 is still 10 times cheaper than GitHub-managed Apple Silicon runners.
Conclusion
Having a fixed price for Cirrus Runners is a great way to save money on your CI/CD infrastructure and just in general have predictable budged.
But it requires keeping the balance between price per minute and queue time. Cirrus Runners Dashboard helps you to keep an eye on this balance
and make sure that you are getting the most out of your Cirrus Runners.