# Autoscaling

> Add burst runners to a GitLab or GitHub Actions runner on Hetzner Cloud or DigitalOcean while CI jobs queue up, and drain them when the load falls, on Hetzner near the end of a paid hour.

Source: https://managerunners.com/docs/manual/autoscaling/

Autoscaling adds runners on Hetzner Cloud or DigitalOcean while one of your GitLab or GitHub Actions runners has few free job slots, and removes them again when the load fits on fewer. The runner you set it up on is the **anchor**. The runners it adds are **burst runners**: Hetzner servers or DigitalOcean droplets registered like the anchor, so they get the same jobs.

- **GitLab:** they are registered with the anchor's GitLab runner authentication token, so GitLab sees them as additional machines of the same runner, with the same tags.
- **GitHub:** they are registered as runners of the anchor's repository or organization, in its runner group, with its labels. Use those labels in your workflows' `runs-on`, so jobs go to the anchor and its burst runners.

Autoscaling never stops the anchor and never scales to zero; you can still pause the anchor by hand or with its schedule, which removes its burst runners. It can be a runner created with Manage Runners or an [external runner](https://managerunners.com/docs/manual/external-runners/), for example your own dedicated server.

## Requirements

- **A Pro or Enterprise plan** for the organization, like [schedules](https://managerunners.com/docs/manual/schedules/). See [Billing and plans](https://managerunners.com/docs/manual/billing/). Without one, the dashboard shows nothing about autoscaling.
- **Metrics from the anchor.** Decisions use the anchor's [Monitoring](https://managerunners.com/docs/manual/metrics/) samples, so turn on **Show in dashboard** for it. Burst runners always have Monitoring.
- **External anchors** need an agent that reports their job limit. For GitLab, that is the agent version 1.1.0 or later with GitLab Runner's metrics endpoint, `listen_address = "127.0.0.1:9252"` in `/etc/gitlab-runner/config.toml`; see [Running jobs on GitLab runners](https://managerunners.com/docs/manual/external-runners/#running-jobs-on-gitlab-runners). For GitHub, that is the agent version 1.2.0 or later, which counts the jobs of the actions runner; see [Running jobs on GitHub runners](https://managerunners.com/docs/manual/external-runners/#running-jobs-on-github-runners). Without a job limit, free job slots cannot be counted and the dashboard says so.
- **GitHub runners created with Manage Runners** before their job counts were added report them once their server is recreated, for example by pausing and resuming them. Until then their jobs are unknown, so a free job slots rule cannot decide.

## Set it up in the dashboard

Runner cards show an **Autoscaling** line next to the schedule. Owners and editors select **Add autoscaling**, or the policy or the pencil next to it, to open the runner's autoscaling page, and choose:

- **When to scale:** **Eager**, **Balanced** or **Conservative**, or **Custom** rules. Below the choice, the page explains what it means for the job slots the runner has now, for example *With 4 job slots now: a burst runner starts when all 4 are busy for 3 minutes.*
- **Maximum burst runners:** 1 to 20. Burst runners are counted until they are deleted, including those still starting or draining.
- **Runners added per step:** 1 to 5, default 1.
- **Remove after low load for (minutes):** how long the load must stay low before one burst runner is removed, default 10.
- **Wait after ready (minutes):** how long to wait after a burst runner became ready before adding more, default 3.
- **Enabled:** for a new policy, whether it starts scaling once saved. An existing policy has **Enable** and **Disable** at the top of the page, and a switch on the card. Disabling drains and deletes every burst runner but keeps the settings.

The ⓘ next to each limit shows a short explanation on hover. Select it to see that explanation together with a longer one.

For an external runner you also describe its burst runners, since Manage Runners does not know how your server is set up: the **Cloud provider**, server type and location, your **Hetzner API token** or **DigitalOcean API token**, concurrency, the SSH keys of the token's Hetzner project or DigitalOcean account to add (listed by name once the token is entered, or with the stored token when you edit the policy), labels, optionally a [Metrics connection](https://managerunners.com/docs/manual/metrics/), and the settings of its platform:

- **GitLab:** the GitLab instance, the runner's **authentication token** (`glrt-…`), the executor and optionally a [cache](https://managerunners.com/docs/manual/caches/). The runner token is checked with GitLab, so your GitLab instance must be reachable over HTTPS at a public address.
- **GitHub:** the repository or organization the runner belongs to, for an organization its runner group, and a **fine-grained personal access token** with the permission a GitHub runner of that scope needs: *Administration* (read and write) for a repository, *Self-hosted runners* (read and write) for an organization. Concurrency is 1 to 32, as for GitHub runners. Enter the runner's own labels, as GitHub gives burst runners the jobs that ask for them. The token is checked with GitHub whenever it, the repository, the organization or the runner group changes. Like on the runner form, the page clears the token after each attempt to save; enter it again to retry.

The provider token is checked with the provider before the policy is saved. For Hetzner you choose one of your organization's [saved Hetzner tokens](https://managerunners.com/docs/manual/hetzner/#saved-hetzner-tokens), by default the one used most recently, or enter a new one, which is saved to your organization, named after the runner unless you name it; burst runners then use the saved token, so replacing it in **Organization settings** reaches them too. The tokens are stored encrypted and never shown again; leave the fields empty later to keep them. Burst runners of a Manage Runners runner copy that runner when they are created: server type, location, Hetzner project or DigitalOcean account, GitLab or GitHub settings with the token, concurrency, SSH keys, labels, cache and Metrics connection.

When DigitalOcean is offered, the page also has a **Cloud provider** choice for the burst runners. For a Manage Runners runner it starts with the runner's own provider. Choose the other one, for example DigitalOcean burst runners for a Hetzner runner, and you enter the burst runners' server type, location, API token and SSH keys of that provider; everything else still comes from the runner. The page shows what the chosen provider bills, for DigitalOcean per second; see [DigitalOcean setup](https://managerunners.com/docs/manual/digitalocean/#billing). A GitHub burst runner gets its own short-lived registration token from GitHub, with the anchor's GitHub token, when its server is set up.

The card shows the policy, for example **Autoscaling · Balanced · up to 3 burst runners**, with the group's job slots, for example **3 of 8 job slots busy · 5 free**, the number of burst runners and the last decision. The autoscaling page also lists the newest 50 events and has **Delete policy**. Events are kept for 90 days; the API and `list_scaling_events` return up to 200 of them.

## Free job slots

A **job slot** is room for one CI job. A runner's job slots are its concurrency; an external runner's job slots are the job limit its agent reports. The agent counts the running jobs of a GitLab runner from GitLab Runner's metrics, and those of a GitHub runner from the actions runner's processes: each runner instance waits for jobs in a listener process, one per job slot, which starts a worker process for each job it runs. The **group** is the anchor and its burst runners that are ready, and its **free job slots** are its job slots minus its running jobs.

The free job slots rule has a number **K** and a duration:

- **Scale out** when at most K job slots are free for the whole duration, for example when all of them are busy (K = 0).
- **Scale in** when, without one burst runner, more than K job slots would still be free for the whole **Remove after low load** time. The jobs of that runner count as moved to the others. A burst runner is also removed when the group ran no job at all for the whole **Remove after low load** time, so a rule whose K is as large as the group's job slots still scales back.

This keeps the number of burst runners from swinging back and forth. A burst runner is added only while at most K job slots are free and the group is not idle. It is removed in one of two cases, and neither can add it straight back:

- More than K job slots would still be free without it, so the group stays above the point where runners are added.
- The group ran no job at all for the whole **Remove after low load** time. While the group stays idle, nothing is added.

## Presets

The presets are one free job slots rule:

| Preset | Adds a burst runner when | For |
| --- | --- | --- |
| Eager | at most 1 job slot is free | 2 minutes |
| Balanced (default) | no job slot is free | 3 minutes |
| Conservative | no job slot is free | 5 minutes |

The page offers only presets that make sense for the anchor: a runner with a single job slot never has more than one free, so Eager would add a burst runner as soon as any job is seen, and it is not offered there. Presets that would behave the same for the anchor are offered once. While an external runner has not reported its job limit, all presets are offered. A policy that already uses a preset keeps showing it; with Eager on a runner with one job slot, its burst runners are removed again once the group has run no job for the **Remove after low load** time.

### Examples

**A runner with 1 job slot**, Balanced: while its job runs for 3 minutes, a burst runner with 1 slot is added. With both busy for another 3 minutes, a second is added, up to the maximum. When the anchor and one burst runner are idle, removing that burst runner still leaves at least one free slot for 10 minutes, so it is removed: on Hetzner in the last 15 minutes of its paid hour, on DigitalOcean at once.

**A runner with 4 job slots**, Balanced: a burst runner is added when all 4 are busy for 3 minutes. With 8 slots and 3 jobs, removing a 4-slot burst runner would leave 1 free slot, more than 0, so it is removed once that has been true for 10 minutes. With Eager, a burst runner is added already when 3 of the 4 are busy for 2 minutes, and removed only when, without it, at least 2 slots would stay free.

## Custom rules

A policy has one to three rules, one per metric:

| Metric | Rule |
| --- | --- |
| Free job slots | Scale out when at most K (0 to 20) job slots are free; scale in when more than K would stay free without one burst runner, or when the group ran no job for the whole **Remove after low load** time. |
| CPU | Average CPU usage of the runners, with a **scale-out** threshold in percent and a lower **scale-in** threshold |
| Memory | Average memory usage of the runners, with a scale-out and a lower scale-in threshold in percent |

Each rule also has a duration of 2 to 60 minutes. In the editor, a free job slots rule reads *Scale out when at most [K] job slots are free for [M] minutes*.

## How decisions are made

Manage Runners checks every policy every 15 seconds. The **group** is the anchor plus its burst runners that are ready and report metrics; burst runners that are still starting count only to wait for them.

- **Scale out** when any rule holds in every minute of its duration: at most K job slots free, or CPU or memory at or above the scale-out threshold. Each report counts for one minute only, and a minute without a report does not count, so a short spike does not count. Nothing is added while the group is idle, whatever the rules say: when the newest report of the anchor and of every ready burst runner says that no job is running. If a runner reports no job count, for example an external GitLab runner without GitLab Runner's metrics endpoint, the group is not known to be idle, so a CPU or memory rule can still add burst runners. Nothing is added either while a burst runner is starting, within **Wait after ready** after the newest became ready, at the maximum, during a wait after a failure or for the cloud provider, or without an active plan.
- **Scale in** when every rule allows it without one burst runner for the whole **Remove after low load** time: more than K job slots would stay free, counting that runner's jobs on the others, or no job ran in the group at all, and CPU and memory would stay below their scale-in thresholds, with the average of the others taken as the average × N ÷ (N − 1). One burst runner is removed at a time, and the load is checked again before the next one. A burst runner is only removed when its own load is known for the whole **Remove after low load** time; otherwise the card says that data is missing.
- **No decisions without metrics:** when the anchor has not reported for about three minutes, nothing is added or removed, except that burst runners are removed when the policy is disabled or the anchor is paused.

## Billing-aware scale-in

Hetzner bills a server for every hour it has started. Removing a Hetzner burst runner just after a new hour began would waste most of that hour, so it is only removed in the **last 15 minutes of its current paid hour**, counted from the time Hetzner created its server. If several qualify, the one closest to the end of its hour goes first; if that is a tie, the one with fewer running jobs. Until one is in its last 15 minutes, the card shows that the load is low and autoscaling is waiting.

DigitalOcean bills droplets per second, so there is no paid hour to wait for: a DigitalOcean burst runner is removed as soon as the load has been low for the **Remove after low load** time. If several qualify, the one with fewer running jobs goes first.

A removed burst runner first takes no new jobs and finishes the running ones, like **Pause after running jobs**, for at most one hour; see [Pause and resume](https://managerunners.com/docs/manual/runners/#pause-and-resume). Then it is deleted completely, with its server, its IP addresses and its entry in the dashboard.

A GitHub burst runner is drained through GitHub: its idle runner instances are removed from GitHub at once, and an instance that is running a job loses its own labels and keeps the server until the job has finished. Until then, GitHub can still give it a job that asks only for the default labels `self-hosted`, the operating system or the architecture, so use one of the anchor's own labels in `runs-on`. A job that is still running after that hour ends when the server is deleted, as on a GitLab burst runner; GitHub then removes that instance once it has been offline for 14 days.

## Burst runners in the dashboard

Burst runners appear after their anchor with a **Burst** badge and show how long they have been running, counted from the time the provider created their server, for example *Running for 1 h 12 min · since 14:02*, or *Starting for 3 min* while they start, and *Draining ·* in front while they are removed. Times are shown in your browser's local time format. The autoscaling page lists the same for each burst runner, oldest first. They can only be deleted: editing, pausing, resuming, duplicating and schedules are not available, also not through the CLI, the API or AI assistants. If you delete one by hand, autoscaling adds another one when the group is busy again. Their servers are in your Hetzner project or DigitalOcean account and billed by the provider like any other runner: in the anchor's project or account when they use the anchor's provider, otherwise in the project or account of the token entered on the autoscaling page.

## GitLab runner managers

This section is about GitLab runners; GitHub burst runners are removed from GitHub when they are drained or deleted.

GitLab shows every machine registered with a runner token as a **runner manager** of that runner, with its own system ID. When a burst runner's server is deleted, Manage Runners removes its runner manager through GitLab's API, so the runner's page in GitLab does not fill up with machines that are gone. Only that machine is removed; the runner, the anchor and the other burst runners stay registered. The same happens when any Manage Runners GitLab runner is paused or deleted, since a resumed runner registers as a new machine.

Manage Runners contacts your GitLab instance for this only over HTTPS and only at public addresses, and tries again for about 20 seconds if GitLab is unavailable. A self-managed GitLab instance that is reachable only from a private network, or a removal that still fails, keeps the old managers until GitLab removes them itself after a week without contact.

## When autoscaling stops

- **Disabling** the policy drains and deletes every burst runner. Enabling it again also forgets earlier failures and ends a wait for the cloud provider.
- **Deleting** the policy, or deleting the anchor, does the same.
- **Pausing** a Manage Runners anchor, for example by its schedule, drains and deletes every burst runner, and nothing is added until it is running again. A burst runner whose creation started just as the anchor began pausing is marked for removal at a later check and then leaves like any other: once it has finished starting, it drains and is deleted.
- **Failures:** when a burst runner cannot be created, or is not ready within 15 minutes, is unknown or fails to register, it is deleted and counts as a failure. The burst runners added in one step count as one failure together. After the first failure Manage Runners waits 5 minutes before adding runners again, after the second 10 minutes; the third failure in a row disables the policy, drains its burst runners, shows the reason on the card and emails the organization's owners and editors once. A refused token, a missing permission or a locked account are failures.
- **Provider shortages** never disable the policy. When the cloud provider has no capacity for the size in the location, an account limit is reached, or the provider or the CI platform is briefly unavailable, the burst runner is deleted without counting as a failure, and Manage Runners waits 5, 10, 20 and then every 30 minutes before trying again, until a burst runner becomes ready. The page shows that autoscaling waits for provider capacity and until when, and the events show when the shortage started or changed. A reached limit also emails the organization's owners and editors once, as only they can raise it.
- **Unknown burst runners**, and burst runners whose creation was interrupted, are never just removed from the dashboard, since their server could keep running. Every Hetzner server Manage Runners creates carries the Hetzner label `managerunners-runner` with the runner's id, and so do its primary IP addresses; every DigitalOcean droplet carries the tag `managerunners-runner:<runner id>`; DigitalOcean burst runners have no reserved IPs, as they use their droplet's own addresses. Manage Runners finds the server by its id or that label or tag and deletes it with its IP addresses. Only the burst runner's own provider token can show that the server is gone; if the provider refuses it, the anchor's or template's token is used only when it finds the labelled server itself. If the cleanup still fails, the card shows that it is blocked and why, the runner keeps counting against the maximum and Manage Runners tries again every minute. Restore the token's access, or delete the server and its IP addresses in your Hetzner project or DigitalOcean account and then the runner in the dashboard.
- **Platform:** an external runner's platform cannot change while it has an autoscaling policy; delete the policy first. Should a policy still have burst runner settings for another platform than its runner, no burst runners are added and the card says to delete the autoscaling policy and create it again.
- **Without an active plan** no burst runners are added. Existing ones start draining, Hetzner ones in the last 15 minutes of their paid hour, DigitalOcean ones right away, and get up to one hour to finish running jobs before they are deleted.

## Limits

- One policy per runner, up to 20 burst runners and up to three rules.
- Burst runners on Hetzner Cloud or DigitalOcean: in the anchor's location when a Manage Runners runner's burst runners use its provider, otherwise in the location chosen on the autoscaling page. On DigitalOcean, each burst runner needs a droplet within the account's [limits](https://managerunners.com/docs/manual/digitalocean/#account-limits) and uses the droplet's own IPv4 and IPv6 addresses, not reserved IPs.
- Decisions use the metrics reported once a minute, so autoscaling reacts within a few minutes, not seconds. It does not look at jobs waiting in GitLab's or GitHub's queue, so it never scales from zero and does not use ephemeral GitHub runners.

## With an AI assistant

The [remote MCP server](https://managerunners.com/docs/manual/connect-to-claude/) has `get_scaling_policy`, `list_scaling_events`, `set_scaling_policy`, `enable_scaling_policy`, `disable_scaling_policy` and `delete_scaling_policy`. They never take tokens: for an external runner without stored tokens, `set_scaling_policy` returns a dashboard link to a page that shows the requested policy and its organization, where you enter the tokens yourself and save exactly that policy. To put a Manage Runners runner's burst runners on the other provider for the first time, enter that provider's token on the runner's autoscaling page; `set_scaling_policy` says so. `list_runners` and `get_runner` show which runner a burst runner belongs to.
