In How Manage Runners works, every runner is one server with a fixed number of job slots. When all of them are busy, jobs wait in GitLab. Autoscaling changes that for GitLab runners: while a runner’s job slots stay full, Manage Runners adds more servers to it, and once the load fits on fewer, it removes them again.
This article explains how that works. The idea: add whole runners that share one GitLab runner, decide from the job slots they report, and remove them without throwing away time you already paid for. The details are about doing that without surprises on your Hetzner or DigitalOcean bill. For GitHub Actions runners, How autoscaling works for GitHub Actions runners tells the same story from GitHub’s side.
Anchors and burst runners
Autoscaling is set up per runner with a scaling policy. The runner that has the policy is the anchor: a runner that Manage Runners created, or an external runner such as your own dedicated server. Autoscaling adds capacity next to the anchor and never stops it, so there is no scale-to-zero.
The servers it adds are burst runners. For a Manage Runners anchor, each one is a copy of the anchor at the moment it is added: server type, location, Hetzner project or DigitalOcean account, GitLab settings, concurrency, SSH keys, labels, cache and Metrics connection. The policy can also put them on the other provider, which Hetzner vs DigitalOcean covers. Manage Runners didn’t set up an external anchor, so its policy describes the burst runners instead: cloud provider, server type, location, provider token, GitLab instance and runner token, executor, concurrency and the rest. Saving the policy checks the SSH keys against the token’s Hetzner project or DigitalOcean account, and a new or changed runner token with GitLab. That catches invalid SSH key selections and rejected runner tokens before any server is provisioned.
A policy has a maximum of 1 to 20 burst runners, a step size of 1 to 5, and rules. Burst runners always have Monitoring on, because decisions depend on their reports. Otherwise they are ordinary runners, shown after their anchor with a Burst badge. You can delete one, but editing, pausing, resuming, duplicating or scheduling it is refused, so the policy and a person don’t fight over a server.
One GitLab runner, many machines
A burst runner is not a new runner in GitLab. It registers with the anchor’s runner authentication token. GitLab allows the same authentication token on several machines: each machine is a runner manager of that one runner, told apart by a system ID that GitLab Runner generates on the host. All managers of a runner get the same jobs, with the same tags, because those live on the runner in GitLab, not on the machine. Nothing changes in your .gitlab-ci.yml.
A removed server should leave GitLab tidy, too. When a burst runner becomes ready, Manage Runners records its server’s GitLab system ID. When the control plane deletes a server whose system ID it recorded, it requests deletion of exactly that runner manager with DELETE /api/v4/runners/managers, using the runner authentication token and the system ID. It’s the request GitLab Runner sends when you unregister one machine, which deletes only that manager, not the runner. The control plane does the same for any GitLab runner whose server it deletes, on a pause too. If GitLab can’t be reached, the manager stays until GitLab’s own stale runner manager cleanup removes it.
When an external anchor’s policy is first saved, or its GitLab host or runner token changes, the control plane calls POST /api/v4/runners/verify to check the runner token. GitLab requires a system ID for it, and the control plane then requests removal of the temporary manager the check registers.
Those two requests are all the control plane sends to GitLab, both with the runner authentication token, on endpoints that GitLab Runner itself uses. Manage Runners has no GitLab account or access token and doesn’t read your projects, pipelines or jobs. Jobs flow only between the servers and GitLab.
The signal: free job slots
A job slot is room for one CI job. A Manage Runners runner has as many as its concurrency, which is also GitLab Runner’s concurrent setting on the server. An external runner reports its limit.
The numbers come from the metrics collector that already runs on each server. Once a minute it reads the running jobs and the concurrent setting from GitLab Runner’s own metrics endpoint, which listens only on the server, and pushes them to Manage Runners with CPU and memory usage.
The group is the anchor plus its burst runners that are ready and reported within about three minutes. With a report from every member, its free job slots are:
free = (anchor slots + Σ slots of ready burst runners) − (running jobs of all of them)
Rules evaluate the past minutes one at a time, as one-minute buckets. A bucket is anchored by an unused report less than 90 seconds old at the bucket’s end, a report is never reused for another bucket, and a member’s reports within one bucket are averaged. The group value for a minute uses the members with a known value for it; without the anchor’s value, that minute can’t satisfy a rule. A rule holds only when it holds in every minute of its duration, so a short spike doesn’t scale anything out.
Scaling out
The main rule has one number, K, and a duration: scale out when at most K job slots are free for M minutes. The presets are this rule with fixed values:
| Preset | Scale out when | For |
|---|---|---|
| Eager | at most 1 job slot is free | 2 minutes |
| Balanced (default) | no job slot is free | 3 minutes |
| Conservative | no job slot is free | 5 minutes |
Custom rules, up to three, one per metric, allow any K from 0 to 20, and CPU or memory with a scale-out and a lower scale-in threshold in percent. Scale-out needs any rule to hold.
Manage Runners checks every policy every 15 seconds. Even when a rule holds, nothing is added while:
- the group is idle: the newest fresh report of the anchor and of each ready burst runner that has one says that no job is running;
- a burst runner is still starting;
- the newest burst runner became ready less than the wait after ready time ago (default 3 minutes), so the figures can catch up;
- the maximum is reached, counting every burst runner that exists, including starting and draining ones;
- the policy is waiting after a failure or for the cloud provider, or the organization’s plan doesn’t include autoscaling.
Idleness comes from each runner’s newest report, not from a minute’s average, which would still contain a job that just ended. A runner that reports no job count, for example an external runner without GitLab Runner’s metrics endpoint, makes the group not known to be idle, so CPU and memory rules still work there.
A scale-out adds the step size, never more than the maximum allows. With the presets, a burst runner is requested 2 to 5 minutes after the slots fill up, at the next check (about every 15 seconds), and then needs a few minutes to boot and register. One that isn’t ready within 15 minutes is deleted and counts as a failure.
Scaling in
Scale-in asks a different question about the same K: would more than K job slots still be free without one burst runner, for the whole remove after low load window (default 10 minutes)? The jobs of that runner count as if they had moved to the others.
Take an anchor with 4 slots on Balanced (K = 0). Its four slots were full for three minutes, so one four-slot burst runner was added. Later, three jobs run across the eight slots. Without the burst runner, 4 − 3 = 1 slot would be free, which is more than 0. Once that has held for 10 minutes, the burst runner can go.
Asking two questions about the same K keeps the number of burst runners from swinging back and forth: added at K or fewer free, removed only when more than K would stay free without it. After a removal, the group is still above the line where runners are added. CPU and memory rules get the same effect from their two thresholds, and while scale-out needs any rule, scale-in needs every rule to allow it.
Load-based scale-in requires the selected runner’s metric values throughout the window, and every configured rule must allow removal. If a rule fails on the known values, the runner stays.
One case needed a second look before release. A free job slots rule whose K is at least the job slots the group keeps can never see “more than K free” without a runner: with K = 1 on a 1-slot anchor, removing the last burst runner leaves at most 1 free slot, which is never more than 1. The last burst runner would stay forever, idle. So a free job slots rule also allows scale-in when the group ran no job for the whole window. That needs a known zero from every runner in every minute: a missing report or an unknown count proves nothing. It can’t undo itself either, since nothing is added while the group is idle.
Removing a runner when its time is paid for
Hetzner bills a server for every hour it has started, from its creation. Removing a burst runner five minutes into an hour throws away 55 paid minutes that the next busy spell might need.
So for load-based scale-in, Manage Runners starts draining at most one ready burst runner a minute, and a Hetzner one only during the last 15 minutes of its current paid hour, counted from the time Hetzner created its server, not from when Manage Runners requested it. When several qualify, the one closest to the end of its hour goes first; on a tie, the one with fewer running jobs. Deletion follows the drain. Failure cleanup and explicit stops, such as disabling the policy, don’t wait for that window.
DigitalOcean bills droplets per second, with a minimum of 60 seconds, so a DigitalOcean burst runner has no paid hour to wait for. It is drained as soon as the load has been low for the window, still at most one a minute, and among several, the one with fewer running jobs goes first.
The dashboard counts a burst runner’s running time from the same moment, for example Running for 1 h 12 min · since 14:02, so it matches what the provider bills. And for Hetzner burst runners, the explanation next to remove after low load warns that a burst runner can stay up longer than that, because it also waits for the end of its hour.
Giving running jobs time to finish
During load-based scale-in, Manage Runners starts a graceful drain instead of deleting the server right away, the same path as Pause after running jobs. The control plane asks the provider to shut the server down gracefully once it is running: an ACPI shutdown on Hetzner, the droplet’s shutdown action on DigitalOcean. On the server, systemd stops GitLab Runner with SIGQUIT, which is GitLab Runner’s graceful shutdown signal: it stops accepting new jobs and exits as soon as the running ones finish. systemd’s defaults would cut that short, so the servers carry drop-ins like these:
# gitlab-runner.service
[Service]
KillSignal=SIGQUIT
KillMode=mixed
TimeoutStopSec=1h
# poweroff.target
[Unit]
JobTimeoutSec=70min
The stop timeout gives jobs up to an hour, and the power-off timeout is raised above it, because systemd would otherwise force the power-off after 30 minutes. Job containers run in a slice that the shutdown doesn’t stop early, so they keep running until GitLab Runner, which waits for them, has stopped.
Once the provider reports the server off, or the one-hour drain deadline passes, Manage Runners requests its deletion. Jobs still running at the deadline can be interrupted. Unlike a paused runner, a burst runner has no next resume, so Manage Runners also deletes its public IP addresses, primary IPs on Hetzner and reserved IPs on DigitalOcean, and requests removal of its runner manager in GitLab. On Hetzner, draining can run past the end of the paid hour, which then bills another one: the cost of giving jobs time to finish.
Keeping the number of servers bounded
Every scale-out creates servers in your Hetzner project or DigitalOcean account. Most of the engineering went into keeping their number where you set it, and keeping track of each one.
Reserve before provisioning. A burst runner is recorded as a reservation of its anchor in the transaction that stores the runner, before Manage Runners asks the provider for a server. That transaction locks the policy and checks again what the decision was based on:
in one transaction, holding the policy's lock:
the policy is the version the decision was made on, and enabled
the anchor is still active
burst runners of this anchor < maximum
store the runner and its reservation
then, outside the transaction:
ask the provider for the server
Changing, disabling or deleting a policy takes the same lock, so a check running while you lower the maximum can’t add one too many. The maximum caps how many burst servers exist at once: a burst runner counts from its reservation until its server is deleted, also while it starts, drains or waits for a blocked cleanup.
Failures back off, then stop. A burst runner that can’t be created, isn’t ready within 15 minutes or fails to register is deleted and counts as a failure; the burst runners of one scale-out step count once. After the first, Manage Runners waits 5 minutes before adding runners, after the second 10. The third in a row disables the policy, drains its burst runners, shows the reason on the card and emails the organization’s owners and editors. A burst runner that becomes ready resets the count.
Provider shortages only back off. When the cloud provider has no capacity for the size in the location, an account limit is reached or its API is briefly unavailable, the burst runner is deleted without counting as a failure. Manage Runners waits 5, 10, 20, then every 30 minutes before trying again, for as long as the shortage lasts, and emails the owners and editors once about a reached limit.
Keeping track of uncertain servers. The autoscaler keeps the cleanup of an uncertain burst runner pending rather than dropping it, since its server could still be running. Manage Runners labels every Hetzner server it creates with the label managerunners-runner and the runner’s ID, and attempts to label its primary IP addresses too. A DigitalOcean droplet carries the tag managerunners-runner:<runner id>; reserved IPs take no tags, so they are found through the droplet they are assigned to or by their recorded addresses. When a creation was interrupted or a state is unknown, Manage Runners looks for the server by its ID or that label or tag and deletes it through the usual path. Only the token the runner was created with can prove that a server is gone: a token of another project or account finds nothing, and “found nothing” proves nothing. If the provider refuses that token, the anchor’s or the policy’s current token is used only when it finds the runner’s labelled resources itself. If the cleanup can’t finish, the card says why, and the runner keeps counting against the maximum.
Turning it off. Disabling or deleting the policy, deleting the anchor, or pausing a Manage Runners anchor, by hand or by its schedule, starts the removal of every burst runner. Disabling and deleting need no plan, since they only remove servers.
Hetzner vs DigitalOcean
Everything above runs the same way on both providers: the signal, the rules, the reservation, the drain and the cleanup don’t depend on where a server runs. The autoscaler knows each provider only by its billing period, and what the provider offers sets a few limits:
| Hetzner Cloud | DigitalOcean | |
|---|---|---|
| Billing | every started hour | per second, at least 60 seconds |
| Load-based scale-in | in the last 15 minutes of the paid hour, closest to its end first | as soon as the load has been low for the window, fewer running jobs first |
| Server types | x86 and Arm | x86 only |
| Runner labels | also put on the server | not put on the droplet |
| Found again by | the label managerunners-runner on the server and its primary IPs |
the tag managerunners-runner:<runner id> on the droplet |
| Public addresses | primary IPv4 and IPv6 | reserved IPv4 and IPv6 |
Scale-in timing. The victim choice waits for the last 15 minutes of a billing period only when the period is longer than those 15 minutes. With a shorter one, removing a runner never leaves much paid time unused, so every ready burst runner qualifies at once. The remove after low load window and the limit of one drain a minute still apply, so DigitalOcean burst runners also leave one at a time. A shorter window can make DigitalOcean burst runners eligible for removal sooner. On Hetzner, a burst runner must also be in the last 15 minutes of its paid hour; if the window ends too late for that, it stays into another paid hour. What you actually save depends on when draining and deletion finish.
Cross-provider burst runners. A Manage Runners anchor’s burst runners use its provider by default. The policy can put them on the other one instead, for example DigitalOcean burst runners for a Hetzner anchor. It then stores the server type, location, API token and SSH keys for that provider, and saving checks that the server type is offered in the location and that the SSH keys belong to the token’s project or account. Everything else still comes from the anchor: the GitLab instance and runner token, executor, concurrency, labels, cache and Metrics connection. Each burst runner follows its own provider: its scale-in timing, its cleanup and its token are that provider’s, not the anchor’s. An external anchor’s policy names the burst runners’ provider directly.
Architecture. DigitalOcean has no Arm droplets. DigitalOcean burst runners of an Arm anchor on Hetzner are x86 servers, and GitLab gives them the same jobs, because tags don’t describe the architecture. A job whose image exists only for Arm fails there. Manage Runners doesn’t stop you from combining the two, so pick a provider that matches the images your jobs use.
Labels. A runner’s labels in Manage Runners become Hetzner server labels, which Hetzner firewalls can select. Droplets get none, only the owner tag. Job routing doesn’t depend on them: tags are set on the runner in GitLab.
Account limits and addresses. Every DigitalOcean burst runner needs a droplet, a reserved IPv4 and a reserved IPv6 within the account’s limits. A creation refused at a limit doesn’t count as a failure: autoscaling waits and tries again, and emails the organization’s owners and editors once. After a droplet is deleted, DigitalOcean can take some minutes to release its reserved IPv4; until it has, the burst runner isn’t deleted yet and keeps counting against the maximum.
Seeing what it does
Every check records its decision and reason; the runner’s card shows the last one with the group’s capacity, for example 3 of 8 job slots busy · 5 free. The autoscaling page explains the setting for the current slots (With 4 job slots now: a burst runner starts when all 4 are busy for 3 minutes) and keeps an event log in sentences built from the rule that fired:
Added burst runner build 2: no job slot was free for 3 minutes
Draining burst runner build 2: without this runner, at least 1 job slot would have been free for the last 10 minutes
Burst runner build 2 was deleted
When nothing happens, the reason says why, for example that a burst runner is starting or the maximum is reached. Each is logged once when it changes, not on every check.
Policies are also in the CLI, for example manage-runners runner autoscaling set --name build --preset balanced --max 3 --yes, where --yes skips the confirmation prompt, and in the MCP server for AI assistants. The MCP tools never take tokens: when an external runner’s policy needs them, the assistant hands you a dashboard link to enter them yourself.
The trade-offs
This design makes choices you will notice:
- Minutes, not seconds. Reports arrive once a minute, a rule must hold for its duration, and a new server takes a few minutes to boot. Autoscaling absorbs busy periods; it doesn’t make one pipeline start faster.
- No queue signal. Manage Runners sees busy job slots, not jobs waiting in GitLab.
- No scale-to-zero. Autoscaling leaves the anchor running; it supplies the signal.
- Whole servers. A burst runner is a full server, billed by the started hour on Hetzner and per second on DigitalOcean. On Hetzner, load-based removal starts only near the end of a paid hour. Its jobs share it, as on the anchor.
GitLab Runner has its own autoscaling, the Docker Autoscaler and Instance executors, successors to the deprecated Docker Machine executor. There, a runner manager that you run creates instances on demand through fleeting plugins for cloud instance groups, with GitLab-maintained plugins for AWS, Google Cloud and Azure, configured in config.toml with settings such as capacity_per_instance, max_use_count and idle policies. Because that runner manager processes the jobs itself, it can size capacity to the volume of jobs, and an instance can serve a single job before it is replaced.
Manage Runners takes another route. The long-lived component that decides is its control plane, not a process on your infrastructure, and each burst runner is a complete GitLab Runner server registered as one more runner manager. There is no autoscaler configuration on the anchor, it works next to an existing external runner, and every burst runner is an ordinary runner you can see and delete. In exchange, it reacts in minutes and scales in whole servers.
The autoscaling manual has every setting and limit, and the CLI reference the runner autoscaling commands. Autoscaling is part of the Pro and Enterprise plans; Autoscale for peaks in the cost calculator estimates what it adds on Hetzner. For GitHub Actions runners, read How autoscaling works for GitHub Actions runners.