## File: README.md # Actions Runner Controller (ARC) [](https://bestpractices.coreinfrastructure.org/projects/6061) [](https://github.com/jonico/awesome-runners) [](https://artifacthub.io/packages/search?repo=actions-runner-controller) ## About Actions Runner Controller (ARC) is a Kubernetes operator that orchestrates and scales self-hosted runners for GitHub Actions. With ARC, you can create runner scale sets that automatically scale based on the number of workflows running in your repository, organization, or enterprise. Because controlled runners can be ephemeral and based on containers, new runner instances can scale up or down rapidly and cleanly. For more information about autoscaling, see ["Autoscaling with self-hosted runners."](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners) You can set up ARC on Kubernetes using Helm, then create and run a workflow that uses runner scale sets. For more information about runner scale sets, see ["Deploying runner scale sets with Actions Runner Controller."](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#runner-scale-set) ## People Actions Runner Controller (ARC) is an open-source project currently developed and maintained in collaboration with the GitHub Actions team, external maintainers @mumoshu and @toast-gear, various [contributors](https://github.com/actions/actions-runner-controller/graphs/contributors), and the [awesome community](https://github.com/actions/actions-runner-controller/discussions). If you think the project is awesome and is adding value to your business, please consider directly sponsoring [community maintainers](https://github.com/sponsors/actions-runner-controller) and individual contributors via GitHub Sponsors. If you are already the employer of one of the contributors, sponsoring via GitHub Sponsors might not be an option. Just support them by other means! See [the sponsorship dashboard](https://github.com/sponsors/actions-runner-controller) for the former and the current sponsors. ## Getting Started To give ARC a try with just a handful of commands, please refer to the [Quickstart guide](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/quickstart-for-actions-runner-controller). For an overview of ARC, please refer to [About ARC](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller). With the introduction of [autoscaling runner scale sets](https://github.com/actions/actions-runner-controller/discussions/2775), the existing [autoscaling modes](./docs/automatically-scaling-runners.md) are now legacy. The legacy modes have certain use cases and will continue to be maintained by the community only. For further information on what is supported by GitHub and what's managed by the community, please refer to [this announcement discussion.](https://github.com/actions/actions-runner-controller/discussions/2775) ### Documentation ARC documentation is available on [docs.github.com](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/quickstart-for-actions-runner-controller). ### Legacy documentation The following documentation is for the legacy autoscaling modes that continue to be maintained by the community: - [Quickstart guide](/docs/quickstart.md) - [About ARC](/docs/about-arc.md) - [Installing ARC](/docs/installing-arc.md) - [Authenticating to the GitHub API](/docs/authenticating-to-the-github-api.md) - [Deploying ARC runners](/docs/deploying-arc-runners.md) - [Adding ARC runners to a repository, organization, or enterprise](/docs/choosing-runner-destination.md) - [Automatically scaling runners](/docs/automatically-scaling-runners.md) - [Using custom volumes](/docs/using-custom-volumes.md) - [Using ARC runners in a workflow](/docs/using-arc-runners-in-a-workflow.md) - [Managing access with runner groups](/docs/managing-access-with-runner-groups.md) - [Configuring Windows runners](/docs/configuring-windows-runners.md) - [Using ARC across organizations](/docs/using-arc-across-organizations.md) - [Using entrypoint features](/docs/using-entrypoint-features.md) - [Deploying alternative runners](/docs/deploying-alternative-runners.md) - [Monitoring and troubleshooting](/docs/monitoring-and-troubleshooting.md) ## Contributing We welcome contributions from the community. For more details on contributing to the project (including requirements), please refer to "[Getting Started with Contributing](https://github.com/actions/actions-runner-controller/blob/master/CONTRIBUTING.md)." ## Troubleshooting We are very happy to help you with any issues you have. Please refer to the "[Troubleshooting](https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md)" section for common issues. --- ## File: charts/actions-runner-controller/docs/UPGRADING.md ## Upgrading This project makes extensive use of CRDs to provide much of its functionality. Helm unfortunately does not support [managing](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) CRDs by design: _The full breakdown as to how they came to this decision and why they have taken the approach they have for dealing with CRDs can be found in [Helm Improvement Proposal 11](https://github.com/helm/community/blob/main/hips/hip-0011.md)_ ``` There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases. ``` Helm will do an initial install of CRDs but it will not touch them afterwards (update or delete). Additionally, because the project leverages CRDs so extensively you **MUST** run the matching controller app container with its matching CRDs i.e. always redeploy your CRDs if you are changing the app version. Due to the above you can't just do a `helm upgrade` to release the latest version of the chart, the best practice steps are recorded below: ## Steps 1. Upgrade CRDs, this isn't optional, the CRDs you are using must be those that correspond with the version of the controller you are installing ```shell # REMEMBER TO UPDATE THE CHART_VERSION TO RELEVANT CHART VERISON!!!! CHART_VERSION=0.18.0 curl -L https://github.com/actions/actions-runner-controller/releases/download/actions-runner-controller-${CHART_VERSION}/actions-runner-controller-${CHART_VERSION}.tgz | tar zxv --strip 1 actions-runner-controller/crds kubectl replace -f crds/ ``` Note that in case you're going to create prometheus-operator `ServiceMonitor` resources via the chart, you'd need to deploy prometheus-operator-related CRDs as well. 2. Upgrade the Helm release ```shell # helm repo [command] helm repo update # helm upgrade [RELEASE] [CHART] [flags] helm upgrade actions-runner-controller \ actions-runner-controller/actions-runner-controller \ --install \ --namespace actions-runner-system \ --version ${CHART_VERSION} ``` --- ## File: charts/actions-runner-controller/README.md ## Docs All additional docs are kept in the `docs/` folder, this README is solely for documenting the values.yaml keys and values ## Values **_The values are documented as of HEAD, to review the configuration options for your chart version ensure you view this file at the relevant [tag](https://github.com/actions/actions-runner-controller/tags)_** > _Default values are the defaults set in the charts `values.yaml`, some properties have default configurations in the code for when the property is omitted or invalid_ | Key | Description | Default | |-----------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------| | `labels` | Set labels to apply to all resources in the chart | | | `replicaCount` | Set the number of controller pods | 1 | | `webhookPort` | Set the containerPort for the webhook Pod | 9443 | | `syncPeriod` | Set the period in which the controller reconciles the desired runners count | 1m | | `enableLeaderElection` | Enable election configuration | true | | `leaderElectionId` | Set the election ID for the controller group | | | `githubEnterpriseServerURL` | Set the URL for a self-hosted GitHub Enterprise Server | | | `githubURL` | Override GitHub URL to be used for GitHub API calls | | | `githubUploadURL` | Override GitHub Upload URL to be used for GitHub API calls | | | `runnerGithubURL` | Override GitHub URL to be used by runners during registration | | | `logLevel` | Set the log level of the controller container | | | `logFormat` | Set the log format of the controller. Valid options are "text" and "json" | text | | `additionalVolumes` | Set additional volumes to add to the manager container | | | `additionalVolumeMounts` | Set additional volume mounts to add to the manager container | | | `authSecret.create` | Deploy the controller auth secret | false | | `authSecret.name` | Set the name of the auth secret | controller-manager | | `authSecret.annotations` | Set annotations for the auth Secret | | | `authSecret.github_app_id` | The ID of your GitHub App. **This can't be set at the same time as `authSecret.github_token`** | | | `authSecret.github_app_installation_id` | The ID of your GitHub App installation. **This can't be set at the same time as `authSecret.github_token`** | | | `authSecret.github_app_private_key` | The multiline string of your GitHub App's private key. **This can't be set at the same time as `authSecret.github_token`** | | | `authSecret.github_token` | Your chosen GitHub PAT token. **This can't be set at the same time as the `authSecret.github_app_*`** | | | `authSecret.github_basicauth_username` | Username for GitHub basic auth to use instead of PAT or GitHub APP in case it's running behind a proxy API | | | `authSecret.github_basicauth_password` | Password for GitHub basic auth to use instead of PAT or GitHub APP in case it's running behind a proxy API | | | `dockerRegistryMirror` | The default Docker Registry Mirror used by runners. | | | `hostNetwork` | The "hostNetwork" of the controller container | false | | `dnsPolicy` | The "dnsPolicy" of the controller container | ClusterFirst | | `image.repository` | The "repository/image" of the controller container | summerwind/actions-runner-controller | | `image.tag` | The tag of the controller container | | | `image.actionsRunnerRepositoryAndTag` | The "repository/image" of the actions runner container | summerwind/actions-runner:latest | | `image.actionsRunnerImagePullSecrets` | Optional image pull secrets to be included in the runner pod's ImagePullSecrets | | | `image.dindSidecarRepositoryAndTag` | The "repository/image" of the dind sidecar container | docker:dind | | `image.pullPolicy` | The pull policy of the controller image | IfNotPresent | | `metrics.serviceMonitor.enable` | Deploy serviceMonitor kind for for use with prometheus-operator CRDs | false | | `metrics.serviceMonitor.interval` | Configure the interval that Prometheus should scrap the controller's metrics | 1m | | `metrics.serviceMonitor.namespace` | Namespace which Prometheus is running in | `Release.Namespace` (the default namespace of the helm chart). | | `metrics.serviceMonitor.timeout` | Configure the timeout the timeout of Prometheus scrapping. | 30s | | `metrics.serviceAnnotations` | Set annotations for the provisioned metrics service resource | | | `metrics.port` | Set port of metrics service | 8443 | | `metrics.proxy.enabled` | Deploy kube-rbac-proxy container in controller pod | true | | `metrics.proxy.image.repository` | The "repository/image" of the kube-proxy container | quay.io/brancz/kube-rbac-proxy | | `metrics.proxy.image.tag` | The tag of the kube-proxy image to use when pulling the container | v0.13.1 | | `metrics.serviceMonitorLabels` | Set labels to apply to ServiceMonitor resources | | | `imagePullSecrets` | Specifies the secret to be used when pulling the controller pod containers | | | `fullnameOverride` | Override the full resource names | | | `nameOverride` | Override the resource name prefix | | | `serviceAccount.annotations` | Set annotations to the service account | | | `serviceAccount.create` | Deploy the controller pod under a service account | true | | `podAnnotations` | Set annotations for the controller pod | | | `podLabels` | Set labels for the controller pod | | | `serviceAccount.name` | Set the name of the service account | | | `securityContext` | Set the security context for each container in the controller pod | | | `podSecurityContext` | Set the security context to controller pod | | | `service.annotations` | Set annotations for the provisioned webhook service resource | | | `service.port` | Set controller service ports | | | `service.type` | Set controller service type | | | `topologySpreadConstraints` | Set the controller pod topologySpreadConstraints | | | `nodeSelector` | Set the controller pod nodeSelector | | | `resources` | Set the controller pod resources | | | `affinity` | Set the controller pod affinity rules | | | `podDisruptionBudget.enabled` | Enables a PDB to ensure HA of controller pods | false | | `podDisruptionBudget.minAvailable` | Minimum number of pods that must be available after eviction | | | `podDisruptionBudget.maxUnavailable` | Maximum number of pods that can be unavailable after eviction. Kubernetes 1.7+ required. | | | `tolerations` | Set the controller pod tolerations | | | `env` | Set environment variables for the controller container | | | `priorityClassName` | Set the controller pod priorityClassName | | | `scope.watchNamespace` | Tells the controller and the github webhook server which namespace to watch if `scope.singleNamespace` is true | `Release.Namespace` (the default namespace of the helm chart). | | `scope.singleNamespace` | Limit the controller to watch a single namespace | false | | `certManagerEnabled` | Enable cert-manager. If disabled you must set admissionWebHooks.caBundle and create TLS secrets manually | true | | `runner.statusUpdateHook.enabled` | Use custom RBAC for runners (role, role binding and service account), this will enable reporting runner statuses | false | | `admissionWebHooks.caBundle` | Base64-encoded PEM bundle containing the CA that signed the webhook's serving certificate | | | `githubWebhookServer.logLevel` | Set the log level of the githubWebhookServer container | | | `githubWebhookServer.logFormat` | Set the log format of the githubWebhookServer controller. Valid options are "text" and "json" | text | | `githubWebhookServer.replicaCount` | Set the number of webhook server pods | 1 | | `githubWebhookServer.useRunnerGroupsVisibility` | Enable supporting runner groups with custom visibility, you also need to set `githubWebhookServer.secret.enabled` to enable this feature. | false | | `githubWebhookServer.enabled` | Deploy the webhook server pod | false | | `githubWebhookServer.queueLimit` | Set the queue size limit in the githubWebhookServer | | | `githubWebhookServer.secret.enabled` | Passes the webhook hook secret to the github-webhook-server | false | | `githubWebhookServer.secret.create` | Deploy the webhook hook secret | false | | `githubWebhookServer.secret.name` | Set the name of the webhook hook secret | github-webhook-server | | `githubWebhookServer.secret.github_webhook_secret_token` | Set the webhook secret token value | | | `githubWebhookServer.imagePullSecrets` | Specifies the secret to be used when pulling the githubWebhookServer pod containers | | | `githubWebhookServer.nameOverride` | Override the resource name prefix | | | `githubWebhookServer.fullnameOverride` | Override the full resource names | | | `githubWebhookServer.serviceAccount.create` | Deploy the githubWebhookServer under a service account | true | | `githubWebhookServer.serviceAccount.annotations` | Set annotations for the service account | | | `githubWebhookServer.serviceAccount.name` | Set the service account name | | | `githubWebhookServer.podAnnotations` | Set annotations for the githubWebhookServer pod | | | `githubWebhookServer.podLabels` | Set labels for the githubWebhookServer pod | | | `githubWebhookServer.podSecurityContext` | Set the security context to githubWebhookServer pod | | | `githubWebhookServer.securityContext` | Set the security context for each container in the githubWebhookServer pod | | | `githubWebhookServer.resources` | Set the githubWebhookServer pod resources | | | `githubWebhookServer.topologySpreadConstraints` | Set the githubWebhookServer pod topologySpreadConstraints | | | `githubWebhookServer.nodeSelector` | Set the githubWebhookServer pod nodeSelector | | | `githubWebhookServer.tolerations` | Set the githubWebhookServer pod tolerations | | | `githubWebhookServer.affinity` | Set the githubWebhookServer pod affinity rules | | | `githubWebhookServer.priorityClassName` | Set the githubWebhookServer pod priorityClassName | | | `githubWebhookServer.terminationGracePeriodSeconds` | Set the githubWebhookServer pod terminationGracePeriodSeconds. Useful when using preStop hooks to drain/sleep. | `10` | | `githubWebhookServer.lifecycle` | Set the githubWebhookServer pod lifecycle hooks | `{}` | | `githubWebhookServer.service.type` | Set githubWebhookServer service type | | | `githubWebhookServer.service.ports` | Set githubWebhookServer service ports | `[{"port":80, "targetPort:"http", "protocol":"TCP", "name":"http"}]` | | `githubWebhookServer.service.loadBalancerSourceRanges` | Set githubWebhookServer loadBalancerSourceRanges for restricting loadBalancer type services | `[]` | | `githubWebhookServer.ingress.enabled` | Deploy an ingress kind for the githubWebhookServer | false | | `githubWebhookServer.ingress.annotations` | Set annotations for the ingress kind | | | `githubWebhookServer.ingress.hosts` | Set hosts configuration for ingress | `[{"host": "chart-example.local", "paths": []}]` | | `githubWebhookServer.ingress.tls` | Set tls configuration for ingress | | | `githubWebhookServer.ingress.ingressClassName` | Set ingress class name | | | `githubWebhookServer.podDisruptionBudget.enabled` | Enables a PDB to ensure HA of githubwebhook pods | false | | `githubWebhookServer.podDisruptionBudget.minAvailable` | Minimum number of pods that must be available after eviction | | | `githubWebhookServer.podDisruptionBudget.maxUnavailable` | Maximum number of pods that can be unavailable after eviction. Kubernetes 1.7+ required. | | | `actionsMetricsServer.logLevel` | Set the log level of the actionsMetricsServer container | | | `actionsMetricsServer.logFormat` | Set the log format of the actionsMetricsServer controller. Valid options are "text" and "json" | text | | `actionsMetricsServer.enabled` | Deploy the actions metrics server pod | false | | `actionsMetricsServer.secret.enabled` | Passes the webhook hook secret to the actions-metrics-server | false | | `actionsMetricsServer.secret.create` | Deploy the webhook hook secret | false | | `actionsMetricsServer.secret.name` | Set the name of the webhook hook secret | actions-metrics-server | | `actionsMetricsServer.secret.github_webhook_secret_token` | Set the webhook secret token value | | | `actionsMetricsServer.imagePullSecrets` | Specifies the secret to be used when pulling the actionsMetricsServer pod containers | | | `actionsMetricsServer.nameOverride` | Override the resource name prefix | | | `actionsMetricsServer.fullnameOverride` | Override the full resource names | | | `actionsMetricsServer.serviceAccount.create` | Deploy the actionsMetricsServer under a service account | true | | `actionsMetricsServer.serviceAccount.annotations` | Set annotations for the service account | | | `actionsMetricsServer.serviceAccount.name` | Set the service account name | | | `actionsMetricsServer.podAnnotations` | Set annotations for the actionsMetricsServer pod | | | `actionsMetricsServer.podLabels` | Set labels for the actionsMetricsServer pod | | | `actionsMetricsServer.podSecurityContext` | Set the security context to actionsMetricsServer pod | | | `actionsMetricsServer.securityContext` | Set the security context for each container in the actionsMetricsServer pod | | | `actionsMetricsServer.resources` | Set the actionsMetricsServer pod resources | | | `actionsMetricsServer.topologySpreadConstraints` | Set the actionsMetricsServer pod topologySpreadConstraints | | | `actionsMetricsServer.nodeSelector` | Set the actionsMetricsServer pod nodeSelector | | | `actionsMetricsServer.tolerations` | Set the actionsMetricsServer pod tolerations | | | `actionsMetricsServer.affinity` | Set the actionsMetricsServer pod affinity rules | | | `actionsMetricsServer.priorityClassName` | Set the actionsMetricsServer pod priorityClassName | | | `actionsMetricsServer.terminationGracePeriodSeconds` | Set the actionsMetricsServer pod terminationGracePeriodSeconds. Useful when using preStop hooks to drain/sleep. | `10` | | `actionsMetricsServer.lifecycle` | Set the actionsMetricsServer pod lifecycle hooks | `{}` | | `actionsMetricsServer.service.type` | Set actionsMetricsServer service type | | | `actionsMetricsServer.service.ports` | Set actionsMetricsServer service ports | `[{"port":80, "targetPort:"http", "protocol":"TCP", "name":"http"}]` | | `actionsMetricsServer.service.loadBalancerSourceRanges` | Set actionsMetricsServer loadBalancerSourceRanges for restricting loadBalancer type services | `[]` | | `actionsMetricsServer.ingress.enabled` | Deploy an ingress kind for the actionsMetricsServer | false | | `actionsMetricsServer.ingress.annotations` | Set annotations for the ingress kind | | | `actionsMetricsServer.ingress.hosts` | Set hosts configuration for ingress | `[{"host": "chart-example.local", "paths": []}]` | | `actionsMetricsServer.ingress.tls` | Set tls configuration for ingress | | | `actionsMetricsServer.ingress.ingressClassName` | Set ingress class name | | | `actionsMetrics.serviceMonitor.enable` | Deploy serviceMonitor kind for for use with prometheus-operator CRDs | false | | `actionsMetrics.serviceMonitor.interval` | Configure the interval that Prometheus should scrap the controller's metrics | 1m | | `actionsMetrics.serviceMonitor.namespace` | Namespace which Prometheus is running in. | `Release.Namespace` (the default namespace of the helm chart). | | `actionsMetrics.serviceMonitor.timeout` | Configure the timeout the timeout of Prometheus scrapping. | 30s | | `actionsMetrics.serviceAnnotations` | Set annotations for the provisioned actions metrics service resource | | | `actionsMetrics.port` | Set port of actions metrics service | 8443 | | `actionsMetrics.proxy.enabled` | Deploy kube-rbac-proxy container in controller pod | true | | `actionsMetrics.proxy.image.repository` | The "repository/image" of the kube-proxy container | quay.io/brancz/kube-rbac-proxy | | `actionsMetrics.proxy.image.tag` | The tag of the kube-proxy image to use when pulling the container | v0.13.1 | | `actionsMetrics.serviceMonitorLabels` | Set labels to apply to ServiceMonitor resources | | --- ## File: contrib/examples/actions-runner/README.md ## Docs All additional docs are kept in the `docs/` folder, this README is solely for documenting the values.yaml keys and values ## Values **_The values are documented as of HEAD, to review the configuration options for your chart version ensure you view this file at the relevent [tag](https://github.com/actions/actions-runner-controller/tags)_** > _Default values are the defaults set in the charts values.yaml, some properties have default configurations in the code for when the property is omitted or invalid_ | Key | Description | Default | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | | `labels` | Set labels to apply to all resources in the chart | | | `replicaCount` | Set the number of runner pods | 1 | | `image.repository` | The "repository/image" of the runner container | summerwind/actions-runner | | `image.tag` | The tag of the runner container | | | `image.pullPolicy` | The pull policy of the runner image | IfNotPresent | | `imagePullSecrets` | Specifies the secret to be used when pulling the runner pod containers | | | `fullnameOverride` | Override the full resource names | | | `nameOverride` | Override the resource name prefix | | | `podAnnotations` | Set annotations for the runner pod | | | `podLabels` | Set labels for the runner pod | | | `podSecurityContext` | Set the security context to runner pod | | | `nodeSelector` | Set the pod nodeSelector | | | `affinity` | Set the runner pod affinity rules | | | `tolerations` | Set the runner pod tolerations | | | `env` | Set environment variables for the runner container | | | `organization` | Github organization where the runner will be registered | test | | `repository` | Github repository where the runner will be registered | | | `runnerLabels` | Labels you want to add in your runner | test | | `autoscaler.enabled` | Enable the HorizontalRunnerAutoscaler, if its enabled then replica count will not be used | true | | `autoscaler.minReplicas` | Minimum no of replicas | 1 | | `autoscaler.maxReplicas` | Maximum no of replicas | 5 | | `autoscaler.scaleDownDelaySecondsAfterScaleOut` | [Anti-Flapping Configuration](https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md#anti-flapping-configuration) | 120 | | `autoscaler.metrics` | [Pull driven scaling](https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md#pull-driven-scaling) | default | | `autoscaler.scaleUpTriggers` | [Webhook driven scaling](https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md#webhook-driven-scaling) | | --- ## File: contrib/README.md The `contrib` directory is the place for sharing various example code for deploying and operating `actions-runner-controller`. Anything contained in this directory is provided as-is. The maintainers of `actions-runner-controller` are not yet committed to provide full support for using, fixing, and enhancing it. However, they will make their best effort to collect feedback from early adopters and advanced users like you, and may eventually consider graduating any of the examples as an official addition to the project. See https://github.com/actions/actions-runner-controller/pull/1375#issuecomment-1258816470 and https://github.com/actions/actions-runner-controller/pull/1559#issuecomment-1258827496 for more context. --- ## File: docs/adrs/2022-10-17-runner-image.md # ADR 2022-10-17: Produce the runner image for the scaleset client **Date**: 2022-10-17 **Status**: Done # Breaking Changes We aim to provide an similar experience (as close as possible) between self-hosted and GitHub-hosted runners. To achieve this, we are making the following changes to align our self-hosted runner container image with the Ubuntu runners managed by GitHub. Here are the changes: - We created a USER `runner(1001)` and a GROUP `docker(123)` - `sudo` has been on the image and the `runner` will be a passwordless sudoer. - The runner binary was placed placed under `/home/runner/` and launched using `/home/runner/run.sh` - The runner's work directory is `/home/runner/_work` - `$HOME` will point to `/home/runner` - The container image user will be the `runner(1001)` The latest Dockerfile can be found at: https://github.com/actions/runner/blob/main/images/Dockerfile # Context users can bring their own runner images, the contract we require is: - It must have a runner binary under `/actions-runner` i.e. `/actions-runner/run.sh` exists - The `WORKDIR` is set to `/actions-runner` - If the user inside the container is root, the environment variable `RUNNER_ALLOW_RUNASROOT` should be set to `1` The existing [ARC runner images](https://github.com/orgs/actions-runner-controller/packages?tab=packages&q=actions-runner) will not work with the new ARC mode out-of-box for the following reason: - The current runner image requires the caller to pass runner configuration info, ex: URL and Config Token - The current runner image has the runner binary under `/runner` which violates the contract described above - The current runner image requires a special entrypoint script in order to work around some volume mount limitation for setting up DinD. Since we expose the raw runner PodSpec to our end users, they can modify the helm `values.yaml` to adjust the runner container to their needs. # Guiding Principles - Build image is separated in two stages. ## The first stage (build) - Reuses the same base image, so it is faster to build. - Installs utilities needed to download assets (`runner` and `runner-container-hooks`). - Downloads the runner and stores it into `/actions-runner` directory. - Downloads the runner-container-hooks and stores it into `/actions-runner/k8s` directory. - You can use build arguments to control the runner version, the target platform and runner container hooks version. Preview (the published runner image might vary): ```Dockerfile FROM mcr.microsoft.com/dotnet/runtime-deps:6.0 as build ARG RUNNER_ARCH="x64" ARG RUNNER_VERSION=2.298.2 ARG RUNNER_CONTAINER_HOOKS_VERSION=0.1.3 RUN apt update -y && apt install curl unzip -y WORKDIR /actions-runner RUN curl -f -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz \ && tar xzf ./runner.tar.gz \ && rm runner.tar.gz RUN curl -f -L -o runner-container-hooks.zip https://github.com/actions/runner-container-hooks/releases/download/v${RUNNER_CONTAINER_HOOKS_VERSION}/actions-runner-hooks-k8s-${RUNNER_CONTAINER_HOOKS_VERSION}.zip \ && unzip ./runner-container-hooks.zip -d ./k8s \ && rm runner-container-hooks.zip ``` ## The main image: - Copies assets from the build stage to `/actions-runner` - Does not provide an entrypoint. The entrypoint should be set within the container definition. Preview: ```Dockerfile FROM mcr.microsoft.com/dotnet/runtime-deps:6.0 WORKDIR /actions-runner COPY --from=build /actions-runner . ``` ## Example of pod spec with the init container copying assets ```yaml apiVersion: v1 kind: Pod metadata: name: spec: containers: - name: runner image: command: ["/runner/run.sh"] volumeMounts: - name: runner mountPath: /runner initContainers: - name: setup image: command: ["sh", "-c", "cp -r /actions-runner/* /runner/"] volumeMounts: - name: runner mountPath: /runner volumes: - name: runner emptyDir: {} ``` --- ## File: docs/adrs/2022-10-27-runnerscaleset-lifetime.md # ADR 2022-10-27: Lifetime of RunnerScaleSet on Service **Date**: 2022-10-27 **Status**: Done ## Context We have created the RunnerScaleSet object and APIs around it on the GitHub Actions service for better support of any self-hosted runner auto-scale solution, like [actions-runner-controller](https://github.com/actions-runner-controller/actions-runner-controller). The `RunnerScaleSet` object will represent a set of homogeneous self-hosted runners to the Actions service job routing system. A `RunnerScaleSet` client (ARC) needs to communicate with the Actions service via HTTP long-poll in a certain protocol to get a workflow job successfully landed on one of its homogeneous self-hosted runners. In this ADR, we discuss the following within the context of actions-runner-controller's new scaling mode: - Who and how to create a RunnerScaleSet on the service? - Who and how to delete a RunnerScaleSet on the service? - What will happen to all the runners and jobs when the deletion happens? ## RunnerScaleSet creation - `AutoScalingRunnerSet` custom resource controller will create the `RunnerScaleSet` object in the Actions service on any `AutoScalingRunnerSet` resource deployment. - The creation is via REST API on Actions service `POST _apis/runtime/runnerscalesets` - The creation needs to use the runner registration token (admin). - `RunnerScaleSet.Name` == `AutoScalingRunnerSet.metadata.Name` - The created `RunnerScaleSet` will only have 1 label and it's the `RunnerScaleSet`'s name - `AutoScalingRunnerSet` controller will store the `RunnerScaleSet.Id` as an annotation on the k8s resource for future lookup. ## RunnerScaleSet modification - When the user patch existing `AutoScalingRunnerSet`'s RunnerScaleSet related properly, ex: `runnerGroupName`, `runnerWorkDir`, the controller needs to make an HTTP PATCH call to the `_apis/runtime/runnerscalesets/2` endpoint in order to update the object on the service. - We will put the deployed `AutoScalingRunnerSet` resource in an error state when the user tries to patch the resource with a different `githubConfigUrl` > Basically, you can't move a deployed `AutoScalingRunnerSet` across GitHub entity, repoA->repoB, repoA->OrgC, etc. > We evaluated blocking the change before instead of erroring at runtime and that we decided not to go down this route because it forces us to re-introduce admission webhooks (require cert-manager). ## RunnerScaleSet deletion - `AutoScalingRunnerSet` custom resource controller will delete the `RunnerScaleSet` object in the Actions service on any `AutoScalingRunnerSet` resource deletion. > `AutoScalingRunnerSet` deletion will contain several steps: > > - Stop the listener app so no more new jobs coming and no more scaling up/down. > - Request scale down to 0 > - Force stop all runners > - Wait for the scale down to 0 > - Delete the `RunnerScaleSet` object from service via REST API - The deletion is via REST API on Actions service `DELETE _apis/runtime/runnerscalesets/1` - The deletion needs to use the runner registration token (admin). The user's `RunnerScaleSet` will be deleted from the service by `DormantRunnerScaleSetCleanupJob` if the particular `AutoScalingRunnerSet` has not connected to the service for the past 7 days. We have a similar rule for self-hosted runners. ## Jobs and Runners on deletion - `RunnerScaleSet` deletion will be blocked if there is any job assigned to a runner within the `RunnerScaleSet`, which has to scale down to 0 before deletion. - Any job that has been assigned to the `RunnerScaleSet` but hasn't been assigned to a runner within the `RunnerScaleSet` will get thrown back to the queue and wait for assignment again. - Any offline runners within the `RunnerScaleSet` will be deleted from the service side. --- ## File: docs/adrs/2022-11-04-crd-api-group-name.md # ADR 2022-11-04: Technical detail about actions-runner-controller repository transfer **Date**: 2022-11-04 **Status**: Done # Context As part of ARC Private Beta: Repository Migration & Open Sourcing Process, we have decided to transfer the current [actions-runner-controller repository](https://github.com/actions-runner-controller/actions-runner-controller) into the [Actions org](https://github.com/actions). **Goals:** - A clear signal that GitHub will start taking over ARC and provide support. - Since we are going to deprecate the existing auto-scale mode in ARC at some point, we want to have a clear separation between the legacy mode (not supported) and the new mode (supported). - Avoid disrupting users as much as we can, existing ARC users will not notice any difference after the repository transfer, they can keep upgrading to the newer version of ARC and keep using the legacy mode. **Challenges** - The original creator's name (`summerwind`) is all over the place, including some critical parts of ARC: - The k8s user resource API's full name is `actions.summerwind.dev/v1alpha1/RunnerDeployment`, renaming it to `actions.github.com` is a breaking change and will force the user to rebuild their entire k8s cluster. - All docker images around ARC (controller + default runner) is published to [dockerhub/summerwind](https://hub.docker.com/u/summerwind) - The helm chart for ARC is currently hosted on [GitHub pages](https://actions-runner-controller.github.io/actions-runner-controller) for https://github.com/actions-runner-controller/actions-runner-controller, moving the repository means we will break users who install ARC via the helm chart # Decisions ## APIs group names for k8s custom resources, `actions.summerwind` or `actions.github` - We will not rename any existing ARC resources API name after moving the repository under Actions org. (keep `summerwind` for old stuff) - For any new resource API we are going to add, those will be named properly under GitHub, ex: `actions.github.com/v1alpha1/AutoScalingRunnerSet` Benefits: - A clear separation from existing ARC: - Easy for the support engineer to triage income tickets and figure out whether we need to support the use case from the user - We won't break existing users when they upgrade to a newer version of ARC after the repository transfer Based on the spike done by `@nikola-jokic`, we have confidence that we can host multiple resources with different API names under the same repository, and the published ARC controller can handle both resources properly. ## ARC Docker images We will not start using the GitHub container registry for hosting ARC images (controller + runner images) right after the repository transfer. But over time, we will start using GHCR for hosting those images along with our deprecation story. ## Helm chart We will recreate the https://github.com/actions-runner-controller/actions-runner-controller repository after the repository transfer. The recreated repository will only contain the helm chart assets which keep powering the https://actions-runner-controller.github.io/actions-runner-controller for users to install ARC via Helm. Long term, we will switch to hosting the helm chart on GHCR (OCI) instead of using GitHub Pages. This will require a one-time change to our users by running `helm repo remove actions-runner-controller` and `helm repo add actions-runner-controller oci://ghcr.io/actions` --- ## File: docs/adrs/2022-12-05-adding-labels-k8s-resources.md # ADR 2022-12-05: Adding labels to our resources **Date**: 2022-12-05 **Status**: Superceded [^1] ## Context users need to provide us with logs so that we can help support and troubleshoot their issues. We need a way for our users to filter and retrieve the logs we need. ## Proposal A good start would be a catch-all label to get all logs that are ARC-related: one of the [recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) is `app.kubernetes.io/part-of` and we can set that for all ARC components to be `actions-runner-controller`. Assuming standard logging that would allow us to get all ARC logs by running ```bash kubectl logs -l 'app.kubernetes.io/part-of=actions-runner-controller' ``` which would be very useful for development to begin with. The proposal is to add these sets of labels to the pods ARC creates: #### controller-manager Labels to be set by the Helm chart: ```yaml metadata: labels: app.kubernetes.io/part-of: actions-runner-controller app.kubernetes.io/component: controller-manager app.kubernetes.io/version: "x.x.x" ``` #### Listener Labels to be set by controller at creation: ```yaml metadata: labels: app.kubernetes.io/part-of: actions-runner-controller app.kubernetes.io/component: runner-scale-set-listener app.kubernetes.io/version: "x.x.x" actions.github.com/scale-set-name: scale-set-name # this corresponds to metadata.name as set for AutoscalingRunnerSet # the following labels are to be extracted by the config URL actions.github.com/enterprise: enterprise actions.github.com/organization: organization actions.github.com/repository: repository ``` #### Runner Labels to be set by controller at creation: ```yaml metadata: labels: app.kubernetes.io/part-of: actions-runner-controller app.kubernetes.io/component: runner app.kubernetes.io/version: "x.x.x" actions.github.com/scale-set-name: scale-set-name # this corresponds to metadata.name as set for AutoscalingRunnerSet actions.github.com/runner-name: runner-name actions.github.com/runner-group-name: runner-group-name # the following labels are to be extracted by the config URL actions.github.com/enterprise: enterprise actions.github.com/organization: organization actions.github.com/repository: repository ``` This would allow us to ask users: > Can you please send us the logs coming from pods labelled 'app.kubernetes.io/part-of=actions-runner-controller'? Or for example if they're having problems specifically with runners: > Can you please send us the logs coming from pods labelled 'app.kubernetes.io/component=runner'? This way users don't have to understand ARC moving parts but we still have a way to target them specifically if we need to. [^1]: Superseded by [ADR 2023-03-14](2023-03-14-adding-labels-k8s-resources.md) --- ## File: docs/adrs/2022-12-27-pick-the-right-runner-to-scale-down.md # ADR 2022-12-27: Pick the right runner to scale down **Date**: 2022-12-27 **Status**: Done ## Context - A custom resource `EphemeralRunnerSet` manage a set of custom resource `EphemeralRunners` - The `EphemeralRunnerSet` has `Replicas` in its `Spec`, and the responsibility of the `EphemeralRunnerSet_controller` is to reconcile a given `EphemeralRunnerSet` to have the same amount of `EphemeralRunners` as the `Spec.Replicas` defined. - This means the `EphemeralRunnerSet_controller` will scale up the `EphemeralRunnerSet` by creating more `EphemeralRunner` in the case of the `Spec.Replicas` is higher than the current amount of `EphemeralRunners`. - This also means the `EphemeralRunnerSet_controller` will scale down the `EphemeralRunnerSet` by finding some existing `EphemeralRunner` to delete in the case of the `Spec.Replicas` is less than the current amount of `EphemeralRunners`. This ADR is about how can we find the right existing `EphemeralRunner` to delete when we need to scale down. ## Current approach 1. `EphemeralRunnerSet_controller` figure out how many `EphemeralRunner` it needs to delete, ex: need to scale down from 10 to 2 means we need to delete 8 `EphemeralRunner` 2. `EphemeralRunnerSet_controller` find all `EphemeralRunner` that is in the `Running` or `Pending` phase. > `Pending` means the `EphemeralRunner` is still probably creating and a runner has not yet configured with the Actions service. > `Running` means the `EphemeralRunner` is created and a runner has probably configured with Actions service, the runner may sit there idle, > or maybe actively running a workflow job. We don't have a clear answer for it from the ARC side. (Actions service knows it for sure) 3. `EphemeralRunnerSet_controller` make an HTTP DELETE request to the Actions service for each `EphemeralRunner` from the previous step and ask the Actions service to delete the runner via `RunnerId`. (The `RunnerId` is generated after the runner registered with the Actions service, and stored on the `EphemeralRunner.Status.RunnerId`) > - The HTTP DELETE request looks like the following: > `DELETE https://pipelines.actions.githubusercontent.com/WoxlUxJHrKEzIp4Nz3YmrmLlZBonrmj9xCJ1lrzcJ9ZsD1Tnw7/_apis/distributedtask/pools/0/agents/1024` > The Actions service will return 2 types of responses: > > 1. 204 (No Content): The runner with Id 1024 has been successfully removed from the service or the runner with Id 1024 doesn't exist. > 2. 400 (Bad Request) with JSON body that contains an error message like `JobStillRunningException`: The service can't remove this runner at this point since it has been > assigned to a job request, the client won't be able to remove the runner until the runner finishes its current assigned job request. 4. `EphemeralRunnerSet_controller` will ignore any deletion error from runners that are still running a job, and keep trying deletion until the amount of `204` equals the amount of `EphemeralRunner` needs to delete. ## The problem with the current approach In a busy `AutoScalingRunnerSet`, the scale up and down may happen all the time as jobs are queued up and jobs finished. We will make way too many HTTP requests to the Actions service and ask it to try to delete a certain runner, and rely on the exception from the service to figure out what to do next. The runner deletion request is not cheap to the service, for synchronization, the `JobStillRunningException` is raised from the DB call for the request. So we are wasting resources on both the Actions service (extra load to the database) and the actions-runner-controller (useless outgoing HTTP requests). In the test ARC that I deployed to Azure, the ARC controller tried to delete RunnerId 12408 for `bbq-beets/ting-test` a total of 35 times within 10 minutes. ## Root cause The `EphemeralRunnerSet_controller` doesn't know whether a given `EphemeralRunner` is actually running a workflow job or not (it only knows the runner is configured at the service), so it can't filter out the `EphemeralRunner`. ## Additional context The legacy ARC's custom resource allows the runner image to leverage the RunnerJobHook feature to update the status of the runner custom resource in K8S (Mark the runner as running workflow run Id XXX). This brings a good value to users as it can provide some insight about which runner is running which job for all the runners in the cluster and it looks pretty close to what we want to fix the [root cause](#root-cause) However, the legacy ARC approach means the service account for running the runner pod needs to have elevated permission to update the custom resource, this would be a big `NO` from a security point of view since we may not trust the code running inside the runner pod. ## Possible Solution The nature of the k8s controller-runtime means we might reconcile the resource base on stale cache data. I think our goal for the solution should be: - Reduce wasteful HTTP requests on a scale-down as much as we can. - We can accept that we might make 1 or 2 wasteful requests to Actions service, but we can't accept making 5/10+ of them. - See if we can meet feature parity with what the RunnerJobHook support with compromise any security concerns. Since the root cause of why the reconciliation can't skip an `EphemeralRunner` is that we don't know whether an `EphemeralRunner` is running a job, a simple thought is how about we somehow attach some info to the `EphemeralRunner` to indicate it's currently running a job? How about we send this info from the service to the auto-scaling-listener via the existing HTTP long-poll and let the listener patch the `EphemeralRunner.Status` to indicate it's running a job? > The listener is normally in a separate namespace with elevated permission and it's something we can trust. Changes: - Introduce a new message type `JobStarted` (in addition to the existing `JobAvailable/JobAssigned/JobCompleted`) on the service side, the message is sent when a runner of the `RunnerScaleSet` get assigned to a job, `RequestId`, `RunnerId`, and `RunnerName` will be included in the message. - Add `RequestId (int)` to `EphemeralRunner.Status`, this will indicate which job the runner is running. - The `AutoScalingListener` will base on the payload of this new message to patch `EphemeralRunners/RunnerName/Status` with the `RequestId` - When `EphemeralRunnerSet_controller` try to find `EphemeralRunner` to delete on a scale down, it will skip any `EphemeralRunner` that has `EphemeralRunner.Status.RequestId` set. - In the future, we can expose more info to this `JobStarted` message and introduce more property under `EphemeralRunner.Status` to reach feature parity with legacy ARC's RunnerJobHook