Transport V2

On a home Wi-Fi link, Transport V2 reduced p99 network latency from 420 ms to 160 ms for a robot running inference in the cloud.

5 min read

Consider a robot that sends camera frames to a cloud model four times a second. Every 250 ms it needs a new chunk of actions. If the model takes 87 ms to compute a reply, that leaves about 163 ms to get the frames to the GPU and the actions back to the robot. What happens when Wi-Fi drops a packet?

In our case for off-board inference, we argued that network variability is an engineering problem we can make progress on. Packet loss is one place to start. A transport can deliver every reply and still leave the robot hesitating while it waits for the next action. How quickly the transport recovers matters most on the slowest round trips.

We rebuilt Dreamscale’s robot-to-cloud transport to recover faster from packet loss. In a four-hour test on home Wi-Fi, the median request saved 7 ms of network time. At p99, network time fell from 420 ms to 160 ms, just inside that 163 ms budget. The robot spent 87% less time waiting for actions.

Bar chart of network time per request. Median: Transport V2 54 ms, Transport V1 61 ms, off-the-shelf WebRTC 104 ms. p99: 160, 420 and 504 ms. p99.9: 389, 872 and 992 ms.
Fig. 01 Round-trip latency after subtracting model compute time. Each transport handled roughly 55,000 three-camera observations over four hours on home Wi-Fi, with a target rate of 4 Hz.

Measuring a 4 Hz control loop

We connected one robot client to a GPU host in US West over home Wi-Fi and cable broadband. Each observation contained three synthetic camera frames sized to match real inputs, about 70 kB in total. The client targeted four observations per second and waited for each reply before sending the next. A stand-in for the model reproduced a compute time of about 87 ms so we could compare transports under the same inference workload.

A home network can change appreciably in an hour. To keep a quiet stretch of Wi-Fi from favoring one transport, we interleaved requests at random offsets. All three transports used the same link during the same four hours.

How often did the robot have to wait?

We measured latency from two perspectives. For each request, we subtracted model compute time from the round trip to get network time. The median, p99 and p99.9 describe a typical request and the thresholds for the slowest 1% and 0.1% of requests. For the robot, we counted the fraction of 250 ms control steps it spent waiting for a late reply.

Medianp99p99.9Time waiting
Transport V254 ms160 ms389 ms1.0%
Transport V161 ms420 ms872 ms7.7%
Off-the-shelf WebRTC104 ms504 ms992 ms9.0%

Every transport answered every request. The difference was when the answers arrived. On V1 the robot spent 7.7% of its control steps waiting. On V2 that fell to 1.0%, with less waiting in each of the four hours. The largest latency improvement came at the far end of the distribution. At p99.9, network time fell from 872 ms to 389 ms.

The managed WebRTC service had the longest tail of the three. Its median network time was about twice V2’s (104 ms versus 54 ms), and its p99 was about 3.2 times V2’s (504 ms versus 160 ms). Moving from that service to V2 reduced time waiting by 89%.

Line chart of the share of requests slower than a given network time, on logarithmic axes. Transport V2's line sits well below the others from about 60 ms onward.
Fig. 02 Each curve shows the fraction of requests that exceeded a given network time. To read p99, start at 1% on the vertical axis and follow it across to each curve. Farther left means a faster p99.

What changes over a full day?

Before the three-way comparison, we ran V1 and a research build of V2 for 24 hours on the same link with the same three-camera workload. V2 spent less time waiting in every hour. Across the day, the share of control steps spent waiting fell from 8.0% to 0.9%. For every 10,000 replies, 685 took longer than 250 ms with V1, compared with 74 with V2.

The gap widened as the link got worse. In V1’s worst hour the robot spent 46% of its time waiting for the next action chunk. On the same link during the same hour, that figure was about 1% for V2.

Bar chart of the share of time spent waiting on a late reply in each of 24 hours. Transport V1 peaks at 46%, 37% and 25% in its worst hours; Transport V2 stays under 6% throughout.
Fig. 03 Each pair of bars covers one hour on the same home Wi-Fi link. Bar height gives the share of control steps the robot spent waiting for a reply.

Where the results apply

These measurements cover one home Wi-Fi network, one internet provider and cloud hosts in US West. We used synthetic frames and simulated model compute throughout. The production build matched the research build in a separate 20-hour run. The comparison with WebRTC covers only the four-hour test on the managed service we used.

Both the link and the observation size affect how much V2 helps. On a quiet wired network, slow round trips are rare and the difference between transports is small. Smaller observations also leave less room for improvement. With two cameras, V2 reduced time waiting by 44%.

Availability

Transport V2 is on for every organization, for every model that connects over QUIC. There is nothing to configure. Update the SDK to get it:

pip install -U dreamscale

DocsGet started