Overview
At Dreamscale Labs, we build cloud inference for robots. That means you can fine-tune and run various frontier robotics models over the cloud with real-time latency (and without expensive local GPUs).
We'd been testing diverse models on diverse embodiments, and had been looking for the next challenge to put the latest VLA/WAM models to the test.
That's when we met Lane from Actor Labs, whose company is deploying autonomy onto heavy machinery, e.g. excavators and wheel loaders. (See below for their π model autonomously dumping dirt into a truck.)
We thought it'd be an interesting and intellectually meaningful exercise to test out different latest models on an excavator they had laying around in the warehouse.
We compare 4 models (Physical Intelligence's π0.5, NVIDIA's GR00T N1.7, Black Forest Labs' Flux 3 Action, and Runway's Praxis-1) on a single task: throw away a lump of cardboard trash into the dumpster.
We collect the data, fine-tune the models, and roll them out via Dreamscale's cloud inference to see how well each performed on controlling an excavator.
The Excavator
Enter the Takeuchi TB20e: a 1930 kg (= 4254 lb) electric mini-excavator used on worksites for digging, lifting, and dumping. Actor had already tapped a Raspberry Pi 5 into the ECU's CAN buses to read and write joint commands to the excavator.
At the end of the day, an excavator is just a robot. It has inputs and outputs, and ways to control it to do meaningful stuff.
Inputs:
- 2x forward-facing GMSL2 cameras attached to the roof of the excavator
- 1x scene Logitech C270 webcam
Outputs (joystick controls):
- Swing
- Boom
- Arm
- Bucket
- Thumb
It's important to note that there is no proprioceptive state. The excavator joints do not come with an encoder (to be fair, they don't need one). Therefore, we take the previous tick's joystick commands as the current state.
Data Collection
To start, we collected 50 episodes of data.
Here's Antoine, our CTO (who has never driven an excavator in his life) collect an episode. This took 70 seconds, but towards the end of data collection, he had improved down to 40 seconds.
What we varied per run:
- Different positions of the cardboard trash
- Different lighting conditions (this was inadvertent, but the sun was setting during data collection)
- Different pieces of cardboard trash (40 episodes on one, 10 episodes on another)
What we kept the same:
- Starting and ending pose (home)
- Same dumping strategy
- Same dumpster position
Once we were done, we uploaded the dataset to Hugging Face in LeRobot v3 format.
Note: the actual cameras recorded at a very high resolution (1920x1080), but the models don't need to see fine details anyway, so we downsampled to 224x224 for training. Also because we were bottlenecked by upload speed.
Training
Once the dataset was in HF, we imported it into Dreamscale, then kicked off serverless fine-tuning jobs using Dreamscale Training. It comes pre-configured with the recommended recipe for each model based on the providers' suggestions, and our prior experiences.
Once training finished, we deployed it on Dreamscale's serving engine, which automatically applied test-time optimizations to speed up inference.
The Moment of Truth
While we were waiting for training to complete, it had become nighttime, so we had to bring out floodlights. This was going to be a real test of scene generalization (none of the dataset included night lighting), and we were curious to see how well the models would perform.
Inference was entirely in the cloud via the Dreamscale platform. After all, a typical excavator does not come with an NVIDIA GPU (though this one had a Thor attached which we didn't use).
Was it going to be fast enough? The answer was yes: for π0.5 at 15 Hz (tick size = 66 ms), we hit median <133 ms obs-to-action latency over satellite internet, with almost all chunks staying under 3 control ticks of inference.
Model Comparison
Can π0.5 excavate?
Yes!
It manages to successfully pick up the cardboard and place it in the dumpster just over half the time. We had even trained it on a preliminary 30-episode dataset, and it still managed to succeed just over a third of the time. Keep in mind that the model has never seen this dark lighting condition, yet it still succeeded!
Can GR00T N1.7 excavate?
No.
We found that GR00T misses the first grasp due to imprecise horizontal swing of the excavator base. It still attempts to close the grip (albeit in mid-air) but doesn't correct itself to swing back towards the correct position. This might be because the dataset does not contain any leftward motion (when not gripping the cardboard).
Can Flux 3 Action excavate?
No.
It swung correctly, but never managed to dip the arm enough to pick up the trash. Why might this be? We investigated the decoded future video latents of the model and saw that F3A does in fact predict the arm moving downwards, but the motion difference is so small that the action outputs are only 10–20% of full throttle.
The excavator doesn't respond to controls below a certain threshold of power, and therefore the model, despite wanting to move the arm slightly lower, could not do so. It's interesting to note that during data collection, the arm was operated in bursts of mostly 100% throttles, and the model has learnt to smooth out actions over a longer horizon.
Can Praxis-1 excavate?
No.
Similar to Flux 3 Action, it swings to the correct position, but doesn't dip far enough to grasp the cardboard. Decoding the future latents here, we notice a similar pattern: the slight dipping motion doesn't meet the threshold for actuation. An interesting observation is how the arm appears to disintegrate further into the video prediction horizon (more than F3A).
Common Observed Failures
We list some common failures observed across the models.
The first and most prevalent is failing to grip the cardboard. In the model's defense, even for a human sitting in the operator's seat it's hard to estimate the correct depth and how far out the arm needs to extend. And while swinging, the arm can obscure the view of the cardboard.
Another interesting behavior is failing to release when hovering over the dumpster. We found that the models want to keep extending the arm upwards (even though it is fully extended). We suspect that this is because a lot of our dataset episodes contained maximally extended arm poses before the grip got released.
Limitations
This was a half-day project, and therefore there are lots of limitations due to time constraint:
- We only collected 50 episodes of data
- We only trained it for max a few hours
- We would've loved to test ablations with different cardboard boxes, different dumpster positions and including/excluding the scene cam
- We also would've loved to test other latest models, e.g. RobbyAnt's Lingbot-VA, NVIDIA's Long-WAM, Verne's V0.5, Perceptron's Isaac 0.5 etc.
A big disclaimer is that the results here are not a full proper comparative evaluation across models and should not be taken as such. Treat it as an interesting experiment of tinkering with robot models on an unfamiliar embodiment.
Conclusion & Takeaways
- An excavator is just a robot: plug in the right inputs and outputs, and you can train a robot model to control it
- It's possible to control robots entirely via cloud inference
- Decoding future video state is useful for debugging WAMs
- π0.5 released in April 2025 (1.5 years ago) and it's still one of the best learners for fine-tuning
A human (Antoine) was able to excavate after minimal training (verbal instructions and a few minutes of self-play) and generalized extremely well. Task completion time improved 2x over time (80 seconds → 40 seconds). Robot models still have a fair way to go to achieve such levels of learning velocity and generalization.
When we think of robotics, we immediately think of humanoids or bimanual arms. We hope our blog inspires more experimentation around robot models for different embodiments. On this theme, we have more interesting use cases of Dreamscale coming soon! (hint: vroom vroom)
We thank Lane at Actor Labs for teaching us how to drive an excavator and fixing electronics that broke along the way.