Serving 256 robots from a single node

Cloud inference for robots only makes economic sense if each GPU serves many robots. We describe how one 8×H100 node serves the π0.5 model to 256 robots, with 99% of requests answered within 50 ms.

13 min read

It is almost trivial to make a robot foundation model faster in the cloud. A data-centre GPU turns an observation into an action several times faster than the computer on board a robot. The hard part is running inference at scale. Cloud inference only makes economic sense if the cost of each GPU is amortized across many robots, which means batching their requests the way any inference server does.

Batching robot inference, however, is not so simple. Throughput has to be high for the unit economics to work, and in LLM serving higher throughput usually costs latency. Robots need actions computed from fresh observations to operate in the real world, so they tolerate very little latency. Each robot also sends several requests a second, so a poorly managed request queue either leaves GPUs underused or lets queueing time blow out. Fleets also run many fine-tunes of a model, and those have to share batches as well.

The work inside a request is also unlike a language model's. A call to a vision-language-action model (VLA) such as π0.5, as we serve it, generates no tokens through the VLM: it is one prefill pass of a large model over about 850 image and text tokens, then ten passes of a smaller action expert that turn noise into 30 actions. Continuous batching and prefill/decode disaggregation still apply, but they have to be rebuilt around that shape.

We studied this with π0.5 on one node of eight H100 GPUs. The budget we set was 99% of requests answered within 50 ms of reaching the node. Our fastest single-robot engine runs a call in 14.8 ms,1 so at four requests a second per robot, a copy of the model on every GPU on the node can serve at most about 135 robots. In practice it served 128 under the deadline. The node described in this post serves 256, with a 99th-percentile latency (p99) of 46.7 to 49.8 ms across three machines, through three key tactics:

  • Disaggregate the call: seven GPUs process images and prompts, and the eighth runs every robot's denoising steps.
  • Step-wise batching the denoising steps across robots (instead of request wise).
  • Quantisation in select areas, and custom kernels.

We also achieve up to 157 robots with up to 24 unique LoRA adapters applied. So multiple fine tunes, let’s say for customers that want to fine tune models for their task can share the same node. We find that multi-LoRA serving is more sensitive to precision changes, so we do not apply FP8.

ConfigurationRobotsRobots per GPU
A full copy of the model on every GPU12816
Call disaggregated, denoising steps batched across robots22027.5
The same, with FP8 in ten layers24030
Custom Kernels25632
Robots on 24 LoRA fine-tunes, no FP815719.6

What a VLA call computes

A π0.5 call has two parts. The first processes the input. A vision encoder, SigLIP, turns three camera images into tokens, and the Gemma-2B language model runs once over those tokens and the text prompt, 848 positions in all. Its output is a KV cache, and the pass that produces it is what they call the prefill.

This is a compute bound process.

The second part produces the actions. It starts from 30 actions of random noise. A smaller transformer with 300M parameters, the action expert, attends to the noisy actions and to the keys and values from the prefill, and predicts how to move the actions toward a clean chunk. It does this ten times, and we call these the denoising steps.

Given the size and dimensions that the action expert operates over, it is neither compute nor memory bound but rather latency bound.

For one robot on an H100 with our inference engine, the prefill takes about 8 ms and the ten denoising steps about 6.

Batching Requests

Batching allows us to attain higher throughput though often at the cost of added inference latency. Robot models not only cannot afford that added latency, but increasing the time each batch takes to process, even if it can process more requests at once, means a request that arrives just after an inference call starts has to wait for the previous call to finish, and the next one it rides. This is queuing latency.

Maximising throughput while minimising both forms of raised latency is therefore an interesting challenge.

Now, a batch makes each robot cheaper only if the GPU has fixed costs to share or idle capacity to fill. If the GPU is already saturated, naive batching may earn very little or even hurt in our case.

The prefill has little of either fixed costs to share or idle capacity, because it is already limited by arithmetic bandwidth at batch 1. Most of its time goes to large matrix multiplications, which with one robot's 848 rows already run at roughly 70% of the H100's nominal peak, so a second robot's rows take almost as long again. The prefill costs 8.2 ms of GPU time per robot alone and 6.8 ms per robot in a call of three, and larger calls gain nothing more.

The denoising step is limited by fixed costs. It pushes only 30 rows through the action expert, and reading the expert's weights at the H100's full memory bandwidth would take about a fifth of a 1.07 ms step. The rest is overhead: a step is a chain of more than a hundred small GPU programs, called kernels, each waiting for the output of the one before, and a matrix multiplication this small takes a few microseconds however few rows it is given.

Batching shares those fixed costs across robots. With 32 robots' rows in the chain a step takes 2.00 ms, less than twice as long for 32 times the work. A robot's ten steps cost 10.7 ms of GPU time alone and 0.63 ms in a batch of 32. (The single-robot engine takes 6.3 ms, using Hazy Research's megakernel design, but cannot batch.)

Move along the chart to compare the two parts at any batch size.

Fig. 01 GPU time per robot as robots share a batch, on one H100. From 1 to 32 robots the ten denoising steps get 17× cheaper per robot. The prefill gains 19% by three robots per call and nothing after. The prefill points were measured with the FP8 layers described below, so they sit slightly under the BF16 figures in the text.

Naive batching on a single node, that is, just increasing batch for each GPU is thus an inefficient use of resources as for the prefill phase the GPU is pinned, and at action denoising it is underutilised.

The two parts therefore need opposite arrangements, and thus naive batching on a single node, that is, just increasing the batch for each GPU is an inefficient use of resources.

Rather, it is better to have one batch as wide as possible for the denoising steps (and more can fit on fewer GPUs), and small calls on as many GPUs as possible for the prefill.

Disaggregating the call.

Naive batching as above is inefficient, but not only does it poorly scale, it bars us from tricks that allow us to achieve significantly lower queueing latency.

We run the prefill on seven GPUs and the denoising steps for all robots on the eighth, which we call the denoising GPU. A robot request’s KV cache from the prefill GPUs is copied to the denoising GPU over NVLink in 82 µs. The split is seven prefill GPUs and one denoising GPU because prefill capacity runs out first. Language-model serving calls this arrangement prefill/decode disaggregation (DistServe, Splitwise).

RobotsSchedulerGPU 0· prefillGPU 1· prefillGPU 2· prefillGPU 3· prefillGPU 4· prefillGPU 5· prefillGPU 6· prefillGPU 7· denoisingKV cache over NVLink,82 µs per robotactions return to the robots RobotsSchedulerGPU 0prefillGPU 1prefillGPU 2prefillGPU 3prefillGPU 4prefillGPU 5prefillGPU 6prefillGPU 7· denoisingKV cacheover NVLink,82 µs per robotactions return to the robots
Fig. 02 The disaggregated node. Requests wait in one queue, and the next free prefill GPU takes up to three of them. Each robot's KV cache is then copied to the denoising GPU, which runs the denoising steps for every robot and returns the actions.

The obvious win for queueing latency comes from each of the 7 prefill GPUs firing out of step with each other, such that requests at the front of queue typically do not wait longer than a 7th of prefill latency before being admitted.

The other wins we break down further below.

Batching robots that are on different denoising steps

Robots reach the denoising GPU at different moments. If a batch had to run its ten steps together, each arriving robot would wait for the running batch and all ten denoising steps to finish.

That is not necessary. A denoising step is the same network evaluated at a different time value, and the time value is an input with a separate entry for each robot. The denoising GPU therefore runs one step at a time over every robot present, up to 32, each at its own step. A robot joins at the next step boundary after its keys and values arrive and leaves after its tenth step. This is called stream batching. Language-model servers do the same at token boundaries, where it is known as continuous batching (Orca).

# One denoising step on the denoising GPU (simplified)
batch = [r for r in slots if r.kv_ready][:32]     # whoever is present, up to 32
v = action_expert(batch, t=[r.step for r in batch], kv=[r.kv for r in batch])
for r, v_r in zip(batch, v):
    # Euler step; dt = -1/10: time runs from 1 (noise) to 0
    r.actions += dt * v_r; r.step += 1
    if r.step == 10: send(r.actions)              # done; the slot is free

The batch size follows from the load. 256 robots making four requests a second need 10,240 robot-steps a second. A 16-robot step takes 1.44 ms, so the GPU runs about 690 steps a second, and 10,240 / 690 is about 15 robots per step. We measured 13 to 17.

Layout
Robots
256
Speed 25× slower
Fig. 03 A simulation of the node. In the flow view each request travels from its robot into one queue, coloured by how long it has waited, then goes to the next free prefill GPU with up to two others and takes a lane on the denoising GPU until its ten steps are done. In the timeline, each block's height is our estimate of how much of the GPU's peak compute that work uses, from its operation count and measured time; with fine-tunes, colours mark which fine-tune each robot runs. The animation and the latencies under it come from a simulation of the node, which lands within 3 ms of our measured runs.

Sharing the denoising GPU costs each robot latency. For the slowest 1% of requests the ten steps take 18 to 20 ms of the 50 ms budget, against 6.3 ms for a robot alone.

Disaggregation and step-level batching together take the node from 128 robots to 220. At that load the seven prefill GPUs are busy about 97% of the time, a level at which queues normally grow out of control.

Why the queue does not blow up at 98% utilisation

A server that is almost always busy normally cannot meet a tail-latency budget. Requests arrive unevenly, and with almost no idle time each burst takes long to work off, so waits grow sharply as utilisation approaches 100%. In the standard model of one server with random arrivals and random service times, the average wait at 98% utilisation is 49 times the service time. A full prefill call takes about 18.5 ms, so that would be most of a second.

In its final configuration, described in the next section, the node runs its prefill GPUs 97 to 99% busy at 256 robots and still meets the budget. There are three reasons.

First, the prefill GPUs get more efficient as the queue grows. A free GPU takes up to three waiting robots in one call, and a call of three uses about a fifth less GPU time per robot than a call of one. So when requests pile up, calls fill, each robot costs less, and the queue drains faster. A GPU that is 98% busy is therefore not 98% of the way to its limit, and the queue still drains at 272 robots.

Second, the queue is shared. A waiting request goes to whichever of the seven GPUs frees up next, so it never sits behind one long call. A call of a given size also always takes the same time, so there are no slow outliers to wait behind.

Third, the robots in our test send evenly. Each sends every 250 ms at its own offset, so requests arrive spread out and not in bursts. This is a property of our test and not of every fleet. When we replaced it with random (Poisson) arrivals at the same average rate, the p99 rose from 49.5 ms to 64.6 ms, and the slowest 1% of requests waited 30 ms for a prefill GPU instead of 11 ms.

Hover a bar for its parts.

Fig. 04 Where the slowest 1% of requests spend their time. A request queues for a prefill GPU, waits behind the call already running on it, then has its prefill call and its ten denoising steps. Top: random offsets, as robots are added. Bottom: the same 256 robots under four arrival patterns. Assigned offsets: the server spaces send times evenly. Site stalls: groups of 32 robots pause for 100 to 250 ms, then send together.

Making each robot cheaper with limited quantisation

With the prefill GPUs full, the node gains robots when each prefill gets cheaper or the rest of the request gets shorter. At 1,024 requests a second, every 0.1 ms removed from a prefill is room for three or four more robots. Most prefill time is matrix multiplication that already runs about as fast as the best 16-bit library kernels on this GPU, so the remaining way to make it cheaper is a smaller number format.

The H100 has about 2x higher arithmetic bandwidth at FP8 than BF16, though FP8 is less accurate.

We tested numerics on 768 cases to decide whether a change is acceptable against openpi's JAX implementation, where our error must not be meaningfully worse than that of openpi's own PyTorch port.

Our use of FP8 passes when it is limited to the MLP matrix multiplications in layers 7 to 16 of Gemma's 18, with both operands in FP8 and the outputs in BF16. This makes the prefill about 9% cheaper per robot and takes the node from 220 robots to 240.

The last step, from 240 robots to 256, comes mostly from four kernel changes. One is in the prefill, where we make one pass fewer over the model's widest activation. The other three shorten each denoising step: an attention kernel written for 30 query tokens, a matrix-multiply algorithm chosen for each exact robot count, and one kernel fewer per layer. Each change takes 0.6 to 2.2 ms off the p99, and the four add up to 5.7 ms.

Achieving batch invariance

Batch invariance is a property of inference systems ensuring that the output for any given request is bitwise identical and independent of the batch size, request order, or concurrent traffic on the server.

Most batched servers lack it, because floating-point addition is not associative and a library chooses the order of its additions from the shape of the whole batch (see our glorious Horace He's post at Thinking Machines). We want it so that a robot's request can be replayed alone with the same result.

We have two strict rules that get us there. First, matrix multiplications use only algorithms that do not split the sum over K, because split-K is where the order of additions comes to depend on batch size. Second, the kernels we wrote ourselves, attention included, split work only by a robot's own inputs and never by the batch. This is easier for us than in the general language-model case, because every robot's request has the same fixed shape.

There is one caveat. The matrix-multiply library is closed source, so the first rule rests on what we observed and not on a guarantee.2 We therefore check every configuration bitwise before it serves, comparing a robot alone against the same robot batched with up to 31 others. At 256 robots the two rules cost nothing we could measure.

Serving many LoRA fine-tunes in the same batch

So far every robot has run the same weights. A node shared by several customers would likely need to run several fine-tunes (as is the nature of robot foundation models at the moment), and if each needed its own batches, the gains above would be lost.

One method is LoRA, which freezes a weight matrix W and learns a low-rank correction to it: y = xW + (xA)B, where A and B are thin matrices, it also makes our life as inference engineers much easier when serving multiple sets of weights. Engines that serve many adapters at once (S-LoRA, Punica, vLLM) keep the adapters out of W, because merging them in would give each fine-tune its own weights and its own batches. Every request in a batch shares the base multiplication xW, and two extra kernels apply the correction: a “shrink” v = xA, and an “expand” y += vB that reads y back and rewrites it, with each row using its own request's A and B (Fig. 05, first view).

However, much to our pain, these engines were built for language-model decoding, where a step has one row per request. A prefill has 848 rows per robot, so rewriting y is another full pass over the activation. A denoising step is a chain of small kernels that wait on one another, and two more per projection lengthen it. vLLM's implementation of Punica's kernels made our prefill 42% slower and a 16-robot denoising step three times slower, and its atomic adds do not fix the order of additions, so batch invariance is lost.

Fig. 05 Three robots, each on a different fine-tune (colours as in Fig. 03), in one batch. First view: the usual path, a shared base multiplication, then a shrink and an expand kernel that use each robot's own A and B. Second view: the shrink still runs, and the expand is folded into the base multiplication as extra columns, with zeros in the other robots' slots.

In the prefill, the expand does not need its own kernel. A matrix multiplication is already a sum over the columns of its input, so the expand folds into the base multiplication if the input is given more columns. We append each robot's v to that robot's rows, in a slot reserved for it, with zeros in every other robot's slot, and stack the B matrices of the robots in the call under W (Fig. 05, second view).

# Three robots on three fine-tunes; T tokens per robot, adapter rank r.
# x: [3T, K].  v: [3T, 3r]: robot i's x @ A_i in slot i, zeros elsewhere.
x_cat = torch.cat([x, v], dim=1)               # [3T, K + 3r]
w_cat = torch.cat([W, B_1, B_2, B_3], dim=0)   # [K + 3r, N]
y = x_cat @ w_cat                              # robot i's rows: x @ W + v_i @ B_i

The zeros keep the robots independent: robot 1's rows meet robot 2's B only as 0 × B, which adds exactly zero, so each robot gets the same bits as it would alone. We use this on the widest projection in each Gemma layer, which accounted for 83% of the expand's time, and the other projections keep separate kernels. It passed the accuracy test with a real adapter.

openpi's fine-tuning recipe puts an adapter on every projection in Gemma and in the action expert, and gives each fine-tune its own vision encoder. In the prefill these add about 17% of GPU time per robot, roughly half for the adapters and half for the encoders, against 50% with vLLM's kernels.

In the denoising steps an adapter costs latency instead of GPU time. The extra arithmetic is negligible, but the shrink and the expand remain as two more dependent kernels per projection in an already long chain. We rewrote them to run on a parallel branch of the graph beside the base multiplication and to add their partial sums in a fixed order, which keeps batch invariance. A 16-robot step takes 4.28 ms with vLLM's kernels and 2.13 ms with ours, against 1.44 ms without adapters, and every robot takes ten steps.

As a result, the limit moves from prefill capacity to the latency of the denoising steps. The p99 passes 50 ms while the prefill GPUs are only 82 to 85% busy. Robots on fine-tunes also lose FP8, which failed the accuracy test with the real adapter in every set of layers we tried. A BF16 prefill costs them 0 to 1 ms of p99, since the prefill is no longer the limit.

With 24 fine-tunes shared among the robots, we measured the limit at 157 robots: the p99 was 48.7 ms at 152 robots and 50.9 ms at 160. Without the adapter on the action expert the p99 is 9 to 10 ms lower, because that adapter runs in every denoising step, and sharing one vision encoder across fine-tunes saves 3 to 4 ms. We have not measured what either does to quality.

Limitations

The 256-robot figure holds for robots that send periodically and independently, and many fleets do not. If all 256 somehow happen to send within the same 2 ms, the p99 is about 229 ms.

What this means for the cost of serving a fleet

At one robot per H100, a whole GPU serves a single robot, so its cost is not amortized across other robots. Too high to be economically sensible to a robot deployment company.

At node scale, the cost per robot falls in proportion to the number of robots sharing each GPU. A copy of the model on every GPU serves 16 robots per H100 within the inference budget, 128 on the node, and cuts the cost per robot about sixteenfold.

Our node runs twice that at 32 per H100, which halves the cost per robot when serving the same set of weights. With multi-LoRA, at 19.6 robots per H100, the cost per robot lands between the two. Both regimes become cost appropriate for mass adoption.

Hover a point for its measured p99.

Fig. 06 Server p99 against the number of robots on one node. A dotted segment ends at the first measured robot count where the queue no longer drains. The shaded bar at 256 is the range across three machines. The all-BF16 line is from the slowest machine (51.0 ms at 220 robots; another measured 47.2 ms).

We measured only π0.5. Many robot foundation models share its structure, a large encoder followed by a short iterative decoder, and we expect the same division to apply.

Notes

  1. One π0.5 call for one robot on one H100, in BF16, with this checkpoint and the same inputs: ours 14.8 ms; torch.compile on a rewrite of the model for the compiler, 23.1 ms; TensorRT, 23.2 ms; openpi's PyTorch port, 32.0 ms; openpi JAX, 45.3 ms. openpi JAX ran at its configured 136 prompt tokens; at our 80 it would be roughly 1 ms faster (an estimate). ↩
  2. cuBLASLt, NVIDIA's matrix-multiply library, is closed source, so the behaviour of its non-split-K algorithms is an observation and not a guarantee. Every one that we ran produced bit-identical rows at every robot count from 1 to 32. ↩

References

  1. Physical Intelligence et al. “π0.5: a Vision-Language-Action Model with Open-World Generalization.” 2025. arxiv.org/abs/2504.16054
  2. Physical Intelligence. openpi. github.com/Physical-Intelligence/openpi
  3. Hazy Research. “Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B.” 2025. hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles
  4. Zhong, Y. et al. “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.” OSDI 2024.
  5. Patel, P. et al. “Splitwise: Efficient Generative LLM Inference Using Phase Splitting.” ISCA 2024.
  6. Yu, G.-I. et al. “Orca: A Distributed Serving System for Transformer-Based Generative Models.” OSDI 2022.
  7. He, H. and Thinking Machines Lab. “Defeating Nondeterminism in LLM Inference.” 2025. thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference
  8. Hu, E. et al. “LoRA: Low-Rank Adaptation of Large Language Models.” 2021. arxiv.org/abs/2106.09685
  9. Sheng, Y. et al. “S-LoRA: Serving Thousands of Concurrent LoRA Adapters.” 2023. arxiv.org/abs/2311.03285
  10. Chen, L. et al. “Punica: Multi-Tenant LoRA Serving.” MLSys 2024.