actions-runner-controller Documentation

File: README.md Copy Chapter

Actions Runner Controller (ARC)

About Copy Chapter

Actions Runner Controller (ARC) is a Kubernetes operator that orchestrates and scales self-hosted runners for GitHub Actions.

With ARC, you can create runner scale sets that automatically scale based on the number of workflows running in your repository, organization, or enterprise. Because controlled runners can be ephemeral and based on containers, new runner instances can scale up or down rapidly and cleanly. For more information about autoscaling, see "Autoscaling with self-hosted runners."

You can set up ARC on Kubernetes using Helm, then create and run a workflow that uses runner scale sets. For more information about runner scale sets, see "Deploying runner scale sets with Actions Runner Controller."

People Copy Chapter

Actions Runner Controller (ARC) is an open-source project currently developed and maintained in collaboration with the GitHub Actions team, external maintainers @mumoshu and @toast-gear, various contributors, and the awesome community.

If you think the project is awesome and is adding value to your business, please consider directly sponsoring community maintainers and individual contributors via GitHub Sponsors.

If you are already the employer of one of the contributors, sponsoring via GitHub Sponsors might not be an option. Just support them by other means!

See the sponsorship dashboard for the former and the current sponsors.

Getting Started Copy Chapter

To give ARC a try with just a handful of commands, please refer to the Quickstart guide.

For an overview of ARC, please refer to About ARC.

With the introduction of autoscaling runner scale sets, the existing autoscaling modes are now legacy. The legacy modes have certain use cases and will continue to be maintained by the community only.

For further information on what is supported by GitHub and what's managed by the community, please refer to this announcement discussion.

Documentation Copy Section

ARC documentation is available on docs.github.com.

Legacy documentation Copy Section

The following documentation is for the legacy autoscaling modes that continue to be maintained by the community:

• Quickstart guide
• About ARC
• Installing ARC
• Authenticating to the GitHub API
• Deploying ARC runners
• Adding ARC runners to a repository, organization, or enterprise
• Automatically scaling runners
• Using custom volumes
• Using ARC runners in a workflow
• Managing access with runner groups
• Configuring Windows runners
• Using ARC across organizations
• Using entrypoint features
• Deploying alternative runners
• Monitoring and troubleshooting

Contributing Copy Chapter

We welcome contributions from the community. For more details on contributing to the project (including requirements), please refer to "Getting Started with Contributing."

Troubleshooting Copy Chapter

We are very happy to help you with any issues you have. Please refer to the "Troubleshooting" section for common issues.

File: charts/actions-runner-controller/docs/UPGRADING.md Copy Chapter

Upgrading Copy Chapter

This project makes extensive use of CRDs to provide much of its functionality. Helm unfortunately does not support managing CRDs by design:

The full breakdown as to how they came to this decision and why they have taken the approach they have for dealing with CRDs can be found in Helm Improvement Proposal 11

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much 
community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community 
consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

Helm will do an initial install of CRDs but it will not touch them afterwards (update or delete).

Additionally, because the project leverages CRDs so extensively you MUST run the matching controller app container with its matching CRDs i.e. always redeploy your CRDs if you are changing the app version.

Due to the above you can't just do a helm upgrade to release the latest version of the chart, the best practice steps are recorded below:

Steps Copy Chapter

1. Upgrade CRDs, this isn't optional, the CRDs you are using must be those that correspond with the version of the controller you are installing

# REMEMBER TO UPDATE THE CHART_VERSION TO RELEVANT CHART VERISON!!!!
CHART_VERSION=0.18.0

curl -L https://github.com/actions/actions-runner-controller/releases/download/actions-runner-controller-${CHART_VERSION}/actions-runner-controller-${CHART_VERSION}.tgz | tar zxv --strip 1 actions-runner-controller/crds

kubectl replace -f crds/

Note that in case you're going to create prometheus-operator ServiceMonitor resources via the chart, you'd need to deploy prometheus-operator-related CRDs as well.

2. Upgrade the Helm release

# helm repo [command]
helm repo update

# helm upgrade [RELEASE] [CHART] [flags]
helm upgrade actions-runner-controller \
  actions-runner-controller/actions-runner-controller \
  --install \
  --namespace actions-runner-system \
  --version ${CHART_VERSION}

File: charts/actions-runner-controller/README.md Copy Chapter

Docs Copy Chapter

All additional docs are kept in the docs/ folder, this README is solely for documenting the values.yaml keys and values

Values Copy Chapter

The values are documented as of HEAD, to review the configuration options for your chart version ensure you view this file at the relevant tag

Default values are the defaults set in the charts values.yaml, some properties have default configurations in the code for when the property is omitted or invalid

File: contrib/examples/actions-runner/README.md Copy Chapter

Docs Copy Chapter

All additional docs are kept in the docs/ folder, this README is solely for documenting the values.yaml keys and values

Values Copy Chapter

The values are documented as of HEAD, to review the configuration options for your chart version ensure you view this file at the relevent tag

Default values are the defaults set in the charts values.yaml, some properties have default configurations in the code for when the property is omitted or invalid

File: contrib/README.md Copy Chapter

The contrib directory is the place for sharing various example code for deploying and operating actions-runner-controller.

Anything contained in this directory is provided as-is. The maintainers of actions-runner-controller are not yet committed to provide
full support for using, fixing, and enhancing it. However, they will make their best effort to collect feedback from early adopters and advanced users like you, and may eventually consider graduating any of the examples as an official addition to the project.

See https://github.com/actions/actions-runner-controller/pull/1375#issuecomment-1258816470 and https://github.com/actions/actions-runner-controller/pull/1559#issuecomment-1258827496 for more context.

File: docs/adrs/2022-10-17-runner-image.md Copy Chapter

ADR 2022-10-17: Produce the runner image for the scaleset client

Date: 2022-10-17

Status: Done

Breaking Changes

We aim to provide an similar experience (as close as possible) between self-hosted and GitHub-hosted runners. To achieve this, we are making the following changes to align our self-hosted runner container image with the Ubuntu runners managed by GitHub.
Here are the changes:

• We created a USER runner(1001) and a GROUP docker(123)
• sudo has been on the image and the runner will be a passwordless sudoer.
• The runner binary was placed placed under /home/runner/ and launched using /home/runner/run.sh
• The runner's work directory is /home/runner/_work
• $HOME will point to /home/runner
• The container image user will be the runner(1001)

The latest Dockerfile can be found at: https://github.com/actions/runner/blob/main/images/Dockerfile

Context

users can bring their own runner images, the contract we require is:

• It must have a runner binary under /actions-runner i.e. /actions-runner/run.sh exists
• The WORKDIR is set to /actions-runner
• If the user inside the container is root, the environment variable RUNNER_ALLOW_RUNASROOT should be set to 1

The existing ARC runner images will not work with the new ARC mode out-of-box for the following reason:

• The current runner image requires the caller to pass runner configuration info, ex: URL and Config Token
• The current runner image has the runner binary under /runner which violates the contract described above
• The current runner image requires a special entrypoint script in order to work around some volume mount limitation for setting up DinD.

Since we expose the raw runner PodSpec to our end users, they can modify the helm values.yaml to adjust the runner container to their needs.

Guiding Principles

• Build image is separated in two stages.

The first stage (build) Copy Chapter

• Reuses the same base image, so it is faster to build.
• Installs utilities needed to download assets (runner and runner-container-hooks).
• Downloads the runner and stores it into /actions-runner directory.
• Downloads the runner-container-hooks and stores it into /actions-runner/k8s directory.
• You can use build arguments to control the runner version, the target platform and runner container hooks version.

Preview (the published runner image might vary):

FROM mcr.microsoft.com/dotnet/runtime-deps:6.0 as build

ARG RUNNER_ARCH="x64"
ARG RUNNER_VERSION=2.298.2
ARG RUNNER_CONTAINER_HOOKS_VERSION=0.1.3

RUN apt update -y && apt install curl unzip -y

WORKDIR /actions-runner
RUN curl -f -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./runner.tar.gz \
    && rm runner.tar.gz

RUN curl -f -L -o runner-container-hooks.zip https://github.com/actions/runner-container-hooks/releases/download/v${RUNNER_CONTAINER_HOOKS_VERSION}/actions-runner-hooks-k8s-${RUNNER_CONTAINER_HOOKS_VERSION}.zip \
    && unzip ./runner-container-hooks.zip -d ./k8s \
    && rm runner-container-hooks.zip

The main image: Copy Chapter

• Copies assets from the build stage to /actions-runner
• Does not provide an entrypoint. The entrypoint should be set within the container definition.

Preview:

FROM mcr.microsoft.com/dotnet/runtime-deps:6.0

WORKDIR /actions-runner
COPY --from=build /actions-runner .

Example of pod spec with the init container copying assets Copy Chapter

apiVersion: v1
kind: Pod
metadata:
  name: <name>
spec:
  containers:
    - name: runner
      image: 
      command: ["/runner/run.sh"]
      volumeMounts:
        - name: runner
          mountPath: /runner
  initContainers:
    - name: setup
      image: 
      command: ["sh", "-c", "cp -r /actions-runner/* /runner/"]
      volumeMounts:
        - name: runner
          mountPath: /runner
  volumes:
    - name: runner
      emptyDir: {}

File: docs/adrs/2022-10-27-runnerscaleset-lifetime.md Copy Chapter

ADR 2022-10-27: Lifetime of RunnerScaleSet on Service

Date: 2022-10-27

Status: Done

Context Copy Chapter

We have created the RunnerScaleSet object and APIs around it on the GitHub Actions service for better support of any self-hosted runner auto-scale solution, like actions-runner-controller.

The RunnerScaleSet object will represent a set of homogeneous self-hosted runners to the Actions service job routing system.

A RunnerScaleSet client (ARC) needs to communicate with the Actions service via HTTP long-poll in a certain protocol to get a workflow job successfully landed on one of its homogeneous self-hosted runners.

In this ADR, we discuss the following within the context of actions-runner-controller's new scaling mode:

• Who and how to create a RunnerScaleSet on the service?
• Who and how to delete a RunnerScaleSet on the service?
• What will happen to all the runners and jobs when the deletion happens?

RunnerScaleSet creation Copy Chapter

• AutoScalingRunnerSet custom resource controller will create the RunnerScaleSet object in the Actions service on any AutoScalingRunnerSet resource deployment.
• The creation is via REST API on Actions service POST _apis/runtime/runnerscalesets
• The creation needs to use the runner registration token (admin).
• RunnerScaleSet.Name == AutoScalingRunnerSet.metadata.Name
• The created RunnerScaleSet will only have 1 label and it's the RunnerScaleSet's name
• AutoScalingRunnerSet controller will store the RunnerScaleSet.Id as an annotation on the k8s resource for future lookup.

RunnerScaleSet modification Copy Chapter

• When the user patch existing AutoScalingRunnerSet's RunnerScaleSet related properly, ex: runnerGroupName, runnerWorkDir, the controller needs to make an HTTP PATCH call to the _apis/runtime/runnerscalesets/2 endpoint in order to update the object on the service.
• We will put the deployed AutoScalingRunnerSet resource in an error state when the user tries to patch the resource with a different githubConfigUrl

  Basically, you can't move a deployed AutoScalingRunnerSet across GitHub entity, repoA->repoB, repoA->OrgC, etc.
  We evaluated blocking the change before instead of erroring at runtime and that we decided not to go down this route because it forces us to re-introduce admission webhooks (require cert-manager).

RunnerScaleSet deletion Copy Chapter

• AutoScalingRunnerSet custom resource controller will delete the RunnerScaleSet object in the Actions service on any AutoScalingRunnerSet resource deletion.

  AutoScalingRunnerSet deletion will contain several steps:

  • Stop the listener app so no more new jobs coming and no more scaling up/down.
  • Request scale down to 0
  • Force stop all runners
  • Wait for the scale down to 0
  • Delete the RunnerScaleSet object from service via REST API

• The deletion is via REST API on Actions service DELETE _apis/runtime/runnerscalesets/1
• The deletion needs to use the runner registration token (admin).

The user's RunnerScaleSet will be deleted from the service by DormantRunnerScaleSetCleanupJob if the particular AutoScalingRunnerSet has not connected to the service for the past 7 days. We have a similar rule for self-hosted runners.

Jobs and Runners on deletion Copy Chapter

• RunnerScaleSet deletion will be blocked if there is any job assigned to a runner within the RunnerScaleSet, which has to scale down to 0 before deletion.
• Any job that has been assigned to the RunnerScaleSet but hasn't been assigned to a runner within the RunnerScaleSet will get thrown back to the queue and wait for assignment again.
• Any offline runners within the RunnerScaleSet will be deleted from the service side.

File: docs/adrs/2022-11-04-crd-api-group-name.md Copy Chapter

ADR 2022-11-04: Technical detail about actions-runner-controller repository transfer

Date: 2022-11-04

Status: Done

Context

As part of ARC Private Beta: Repository Migration & Open Sourcing Process, we have decided to transfer the current actions-runner-controller repository into the Actions org.

Goals:

• A clear signal that GitHub will start taking over ARC and provide support.
• Since we are going to deprecate the existing auto-scale mode in ARC at some point, we want to have a clear separation between the legacy mode (not supported) and the new mode (supported).
• Avoid disrupting users as much as we can, existing ARC users will not notice any difference after the repository transfer, they can keep upgrading to the newer version of ARC and keep using the legacy mode.

Challenges

• The original creator's name (summerwind) is all over the place, including some critical parts of ARC:
  • The k8s user resource API's full name is actions.summerwind.dev/v1alpha1/RunnerDeployment, renaming it to actions.github.com is a breaking change and will force the user to rebuild their entire k8s cluster.
  • All docker images around ARC (controller + default runner) is published to dockerhub/summerwind
• The helm chart for ARC is currently hosted on GitHub pages for https://github.com/actions-runner-controller/actions-runner-controller, moving the repository means we will break users who install ARC via the helm chart

Decisions

APIs group names for k8s custom resources, `actions.summerwind` or `actions.github` Copy Chapter

• We will not rename any existing ARC resources API name after moving the repository under Actions org. (keep summerwind for old stuff)
• For any new resource API we are going to add, those will be named properly under GitHub, ex: actions.github.com/v1alpha1/AutoScalingRunnerSet

Benefits:

• A clear separation from existing ARC:
  • Easy for the support engineer to triage income tickets and figure out whether we need to support the use case from the user
• We won't break existing users when they upgrade to a newer version of ARC after the repository transfer

Based on the spike done by @nikola-jokic, we have confidence that we can host multiple resources with different API names under the same repository, and the published ARC controller can handle both resources properly.

ARC Docker images Copy Chapter

We will not start using the GitHub container registry for hosting ARC images (controller + runner images) right after the repository transfer.

But over time, we will start using GHCR for hosting those images along with our deprecation story.

Helm chart Copy Chapter

We will recreate the https://github.com/actions-runner-controller/actions-runner-controller repository after the repository transfer.

The recreated repository will only contain the helm chart assets which keep powering the https://actions-runner-controller.github.io/actions-runner-controller for users to install ARC via Helm.

Long term, we will switch to hosting the helm chart on GHCR (OCI) instead of using GitHub Pages.

This will require a one-time change to our users by running
helm repo remove actions-runner-controller and helm repo add actions-runner-controller oci://ghcr.io/actions

File: docs/adrs/2022-12-05-adding-labels-k8s-resources.md Copy Chapter

ADR 2022-12-05: Adding labels to our resources

Date: 2022-12-05

Status: Superceded [^1]

Context Copy Chapter

users need to provide us with logs so that we can help support and troubleshoot their issues. We need a way for our users to filter and retrieve the logs we need.

Proposal Copy Chapter

A good start would be a catch-all label to get all logs that are
ARC-related: one of the recommended labels
is app.kubernetes.io/part-of and we can set that for all ARC components
to be actions-runner-controller.

Assuming standard logging that would allow us to get all ARC logs by running

kubectl logs -l 'app.kubernetes.io/part-of=actions-runner-controller'

which would be very useful for development to begin with.

The proposal is to add these sets of labels to the pods ARC creates:

controller-manager

Labels to be set by the Helm chart:

metadata:
  labels:
    app.kubernetes.io/part-of: actions-runner-controller
    app.kubernetes.io/component: controller-manager
    app.kubernetes.io/version: "x.x.x"

Listener

Labels to be set by controller at creation:

metadata:
  labels:
    app.kubernetes.io/part-of: actions-runner-controller
    app.kubernetes.io/component: runner-scale-set-listener
    app.kubernetes.io/version: "x.x.x"
    actions.github.com/scale-set-name: scale-set-name # this corresponds to metadata.name as set for AutoscalingRunnerSet

    # the following labels are to be extracted by the config URL
    actions.github.com/enterprise: enterprise
    actions.github.com/organization: organization
    actions.github.com/repository: repository

Runner

Labels to be set by controller at creation:

metadata:
  labels:
    app.kubernetes.io/part-of: actions-runner-controller
    app.kubernetes.io/component: runner
    app.kubernetes.io/version: "x.x.x"
    actions.github.com/scale-set-name: scale-set-name # this corresponds to metadata.name as set for AutoscalingRunnerSet
    actions.github.com/runner-name: runner-name
    actions.github.com/runner-group-name: runner-group-name

    # the following labels are to be extracted by the config URL
    actions.github.com/enterprise: enterprise
    actions.github.com/organization: organization
    actions.github.com/repository: repository

This would allow us to ask users:

Can you please send us the logs coming from pods labelled 'app.kubernetes.io/part-of=actions-runner-controller'?

Or for example if they're having problems specifically with runners:

Can you please send us the logs coming from pods labelled 'app.kubernetes.io/component=runner'?

This way users don't have to understand ARC moving parts but we still have a
way to target them specifically if we need to.

[^1]: Superseded by ADR 2023-03-14

File: docs/adrs/2022-12-27-pick-the-right-runner-to-scale-down.md Copy Chapter

ADR 2022-12-27: Pick the right runner to scale down

Date: 2022-12-27

Status: Done

Context Copy Chapter

• A custom resource EphemeralRunnerSet manage a set of custom resource EphemeralRunners
• The EphemeralRunnerSet has Replicas in its Spec, and the responsibility of the EphemeralRunnerSet_controller is to reconcile a given EphemeralRunnerSet to have
  the same amount of EphemeralRunners as the Spec.Replicas defined.
• This means the EphemeralRunnerSet_controller will scale up the EphemeralRunnerSet by creating more EphemeralRunner in the case of the Spec.Replicas is higher than
  the current amount of EphemeralRunners.
• This also means the EphemeralRunnerSet_controller will scale down the EphemeralRunnerSet by finding some existing EphemeralRunner to delete in the case of
  the Spec.Replicas is less than the current amount of EphemeralRunners.

This ADR is about how can we find the right existing EphemeralRunner to delete when we need to scale down.

Current approach Copy Chapter

1. EphemeralRunnerSet_controller figure out how many EphemeralRunner it needs to delete, ex: need to scale down from 10 to 2 means we need to delete 8 EphemeralRunner

2. EphemeralRunnerSet_controller find all EphemeralRunner that is in the Running or Pending phase.

   Pending means the EphemeralRunner is still probably creating and a runner has not yet configured with the Actions service.
   Running means the EphemeralRunner is created and a runner has probably configured with Actions service, the runner may sit there idle,
   or maybe actively running a workflow job. We don't have a clear answer for it from the ARC side. (Actions service knows it for sure)

3. EphemeralRunnerSet_controller make an HTTP DELETE request to the Actions service for each EphemeralRunner from the previous step and ask the Actions service to delete the runner via RunnerId.
   (The RunnerId is generated after the runner registered with the Actions service, and stored on the EphemeralRunner.Status.RunnerId)

   • The HTTP DELETE request looks like the following:
     DELETE https://pipelines.actions.githubusercontent.com/WoxlUxJHrKEzIp4Nz3YmrmLlZBonrmj9xCJ1lrzcJ9ZsD1Tnw7/_apis/distributedtask/pools/0/agents/1024
     The Actions service will return 2 types of responses:

   1. 204 (No Content): The runner with Id 1024 has been successfully removed from the service or the runner with Id 1024 doesn't exist.
   2. 400 (Bad Request) with JSON body that contains an error message like JobStillRunningException: The service can't remove this runner at this point since it has been
      assigned to a job request, the client won't be able to remove the runner until the runner finishes its current assigned job request.

4. EphemeralRunnerSet_controller will ignore any deletion error from runners that are still running a job, and keep trying deletion until the amount of 204 equals the amount of
   EphemeralRunner needs to delete.

The problem with the current approach Copy Chapter

In a busy AutoScalingRunnerSet, the scale up and down may happen all the time as jobs are queued up and jobs finished.

We will make way too many HTTP requests to the Actions service and ask it to try to delete a certain runner, and rely on the exception from the service to figure out what to do next.

The runner deletion request is not cheap to the service, for synchronization, the JobStillRunningException is raised from the DB call for the request.

So we are wasting resources on both the Actions service (extra load to the database) and the actions-runner-controller (useless outgoing HTTP requests).

In the test ARC that I deployed to Azure, the ARC controller tried to delete RunnerId 12408 for bbq-beets/ting-test a total of 35 times within 10 minutes.

Root cause Copy Chapter

The EphemeralRunnerSet_controller doesn't know whether a given EphemeralRunner is actually running a workflow job or not
(it only knows the runner is configured at the service), so it can't filter out the EphemeralRunner.

Additional context Copy Chapter

The legacy ARC's custom resource allows the runner image to leverage the RunnerJobHook feature to update the status of the runner custom resource in K8S (Mark the runner as running workflow run Id XXX).

This brings a good value to users as it can provide some insight about which runner is running which job for all the runners in the cluster and it looks pretty close to what we want to fix the root cause

However, the legacy ARC approach means the service account for running the runner pod needs to have elevated permission to update the custom resource,
this would be a big NO from a security point of view since we may not trust the code running inside the runner pod.

Possible Solution Copy Chapter

The nature of the k8s controller-runtime means we might reconcile the resource base on stale cache data.

I think our goal for the solution should be:

• Reduce wasteful HTTP requests on a scale-down as much as we can.
• We can accept that we might make 1 or 2 wasteful requests to Actions service, but we can't accept making 5/10+ of them.
• See if we can meet feature parity with what the RunnerJobHook support with compromise any security concerns.

Since the root cause of why the reconciliation can't skip an EphemeralRunner is that we don't know whether an EphemeralRunner is running a job,
a simple thought is how about we somehow attach some info to the EphemeralRunner to indicate it's currently running a job?

How about we send this info from the service to the auto-scaling-listener via the existing HTTP long-poll
and let the listener patch the EphemeralRunner.Status to indicate it's running a job?

The listener is normally in a separate namespace with elevated permission and it's something we can trust.

Changes:

• Introduce a new message type JobStarted (in addition to the existing JobAvailable/JobAssigned/JobCompleted) on the service side, the message is sent when a runner of the RunnerScaleSet get assigned to a job,
  RequestId, RunnerId, and RunnerName will be included in the message.
• Add RequestId (int) to EphemeralRunner.Status, this will indicate which job the runner is running.
• The AutoScalingListener will base on the payload of this new message to patch EphemeralRunners/RunnerName/Status with the RequestId
• When EphemeralRunnerSet_controller try to find EphemeralRunner to delete on a scale down, it will skip any EphemeralRunner that has EphemeralRunner.Status.RequestId set.
• In the future, we can expose more info to this JobStarted message and introduce more property under EphemeralRunner.Status to reach feature parity with legacy ARC's RunnerJobHook