Production inference demand rises and falls in a daily wave. We built a capacity controller that reallocates GPUs between production and research so production tracks demand without over-provisioning. Using queueing theory we optimized allocations, leading to more GPUs for research overnight and shorter queue waits all day.

In a previous post we described how we use Kueue to lend idle GPUs across research initiatives. Here we focus on reclaiming production GPUs for research during off-peak hours, then returning them before the morning peak.

Production inference demand moves in cycles. Even with a global user base, traffic concentrates around North American working hours: it climbs through the morning, peaks around 9am ET and bottoms out around 8pm ET. The trough can be less than half the peak.

That creates a familiar dilemma for every AI company. Provision for peak demand and most GPUs sit idle every night. Provision for trough demand and queues blow out every morning.

Instead, the production fleet should follow the demand cycle: grow into the morning peak, shrink into the evening trough and lend whatever it isn't using to research.

We built a lightweight capacity controller to do this reallocation (internally called deckard, after the Blade Runner protagonist who relentlessly reclaims replicants). It is deliberately narrow, managing exactly two things:

Rather than re-deciding allocations continuously, the controller applies a small set of time windows, each with its own pre-computed schedule:

Windows are coarse on purpose, because moving GPUs between clusters is expensive: draining and tearing a node down on one side and standing it back up on the other takes 20 to 60 minutes on our cloud provider.

It's fair to ask why we make this hard for ourselves and shift capacity between clusters. If research and production shared one multi-tenant Kubernetes cluster, transferring a GPU allocation between them would be a scheduling decision rather than a physical transfer, much like Kueue already lends idle GPUs within a single cluster in the previous post.

We keep the environments in separate clusters anyway. The isolation gives us:

• Blast-radius containment. A runaway research job, or a single overly broad ClusterRole, can't take down customer-facing inference.
• Independent infrastructure. Separate clusters can run different Kubernetes versions, GPU drivers and networking stacks, so we can test risky infra upgrades on research without exposing production. We've needed this when a driver or networking change broke on certain versions, or when we want to run a bleeding-edge PyTorch to take advantage of training performance improvements.

Because transfers are slow, the controller predicts the wave for a given window and moves capacity ahead of demand.

A Crash Course in Queueing Theory

So how many GPUs do we need to service requests? One approach is to use queueing theory. The field predates digital computing. In 1909, the Danish mathematician Agner Krarup Erlang studied telephone switchboards to figure out how many circuits were needed so callers rarely hear a busy signal.

Translating the definitions to our domain, there are four key variables:

• arrival_rate: how fast new generations arrive (requests / second).
• service_rate: how fast one GPU worker serves requests (requests / second). (So average runtime is 1 / service_rate.)
• num_servers: how many GPU workers we run.
• traffic_intensity: how "busy" the system is on average: traffic_intensity = arrival_rate / (num_servers * service_rate).

A motivating example: suppose one GPU worker completes 1 request/sec (service_rate = 1) and you run 100 GPUs for arrival_rate = 95, so traffic_intensity = 0.95. Now say you get a short burst to 110 req/sec for a couple of minutes (or a few GPUs temporarily drain/restart). Backlog accumulates at ~10 req/sec during the burst. Even after demand returns to 95, you only "catch up" at 5 req/sec (100 served − 95 arriving), so that backlog can linger and inflate tail latency far longer than the burst itself.

It gets worse the closer traffic intensity gets to 1. Write ρ for traffic_intensity. Under Erlang-C assumptions, once every GPU is busy each extra queued job is ρ times as likely as the one before it, so the expected backlog is the geometric series ρ + ρ² + ρ³ + ... = ρ / (1 − ρ). As ρ approaches 1 the denominator vanishes and the backlog diverges.

By Little's Law (jobs in system = arrival rate × time in system), waiting time carries the same factor: expected wait scales like 1 / (1 − ρ) times the service time. Headroom is 1 − ρ, so every halving of it doubles the wait: roughly 10× at 90% utilization, 20× at 95%, 50× at 98%.

So we size capacity so that a high percentile of requests (we target the p98) stays within each queue's latency target. Once you can predict arrival_rate and measure service_rate, there are several ways to choose num_servers. We looked at three.

1. Proportional scaling. Scale GPUs linearly with traffic: double the requests, double the GPUs. It's intuitive but wrong because it holds utilization constant across queues of every size. A big queue tolerates much higher utilization than a small one for the same tail latency.

2. Square Root Staffing. A rule of thumb is to calculate num_servers ≈ offered_load + beta * sqrt(offered_load), where offered_load = arrival_rate / service_rate and beta is a small safety factor.

Why the square root? Arrivals over a short window are approximately Poisson, and a Poisson distribution's standard deviation is the square root of its mean. So the random fluctuation you need to buffer scales like sqrt(offered_load).

This also explains why large pools are more efficient than small ones. The buffer as a fraction of the mean shrinks like 1/sqrt(offered_load), so small queues need proportionally more headroom to hit the same tail-latency goal.

3. Marginal gain on top of Erlang-C. Square Root Staffing tells you how to size one queue, but we have many models sharing one finite reservation. The question becomes: which queue benefits most from the next GPU? Marginal gain answers it greedily: score every queue by how much one more GPU would shrink its p98 wait, hand the GPU to the biggest winner, then recompute and repeat. The effect is to pour capacity into whichever queue is closest to its latency cliff.

Inside the allocate Command

To allocate capacity we built a simple CLI where a user or CI process can run deckard allocate. Running it turns all of this into the schedule files the controller applies — and because the output is just declarative YAML in git, the entire review is a plain git diff. For each time window it:

The output is one declarative file per window, which CI applies on merge and hourly:

# Generate every window's schedule from 7 days of real demand
deckard allocate

# Review what would change before anything is applied
git diff live-schedules/

# live-schedules/peak-weekday-day.yaml  (generated)
workloads:
  task-worker-2x-h100-<model>:
    prod-a: 48
    prod-b: 36
compute:
  a3-megagpu-8g:
    us-region-1:
      prod-a:
        counts: [80, 64, 0]

There are knobs for the real world, too. We can size against inflated demand for a known spike (deckard allocate --scale-up 1.1 budgets for +10%), or override a single model's expected rate. That's how we pre-scaled capacity ahead of timed launches like our Gen:48 weekend competition.

Once we've determined the minimum capacity needed to meet our SLOs, the controller pulls GPUs out of production as demand falls toward the evening trough. That freed idle capacity is what the Kueue system lends to research and what it reclaims when the morning peak returns.

That handoff doesn't impact large training jobs because research treats the swappable nodes as preemptible. Each node that can move carries a label marking it ephemeral (runwayml.com/compute-shared-with-external-clusters=true); the rest are static, a research-only floor the controller never touches.

In Kueue, jobs on the shared queue fill the ephemeral nodes first and run there as preemptible tenants, while each team's reserved quota stays on static nodes. When production reclaims capacity, the reserved queues take their quota back, the ephemeral jobs are evicted and the node is drained (force-deleting pods so a long training job's PodDisruptionBudget can't hold the transfer hostage) before it is torn down and rebuilt on the production side. A slow transfer then costs research only an interrupted, requeued job instead of a corrupted run.

There is no tradeoff: by sizing production tightly against real demand, we run with fewer GPUs in production and shorter queue waits for users. The night we first watched production drain an entire idle superblock down to zero GPUs, handing all of it to research, was a small celebration on the team.

What We've Learned

Coarse and reviewable beats continuous and clever, for now. Discrete time windows applied from CI have captured most of the available savings with far fewer failure modes than a fully autonomous controller. We'll reach for continuous rebalancing when the remaining gain clearly justifies the added risk.

One top-down controller, many clusters. Operating across multiple clusters and clouds, a single controller that does the right thing globally beats a swarm of per-cluster autoscalers each with a partial view.