Building reliable physical AI systems requires an edge-centric, cloud-hybrid architecture, and here is why:
- Decisions need to be made in real time and reliably
- Learning from the field is necessary
- Systems must work within fixed hardware and environmental constraints
I’ve seen companies waste infra budgets and engineering resources building their stack just to move data to the cloud. They still approach it like a classic IoT deployment: store everything first and figure out insights later. That approach gets expensive with larger streams and becomes too slow for real-time decisions.
Physical AI systems are different, and they can generate massive amounts of raw data that is messy, noisy, hardware-specific, context-dependent, and generally less useful in its raw form. Take satellite imagery: raw L0 data isn’t ready for downstream use; it requires many processing steps, including geo-registration and distortion correction, before it can be fed to models or used in geospatial applications.
Video data is another example as fleets grow. Sending every camera feed to the cloud for processing isn’t economically viable, and network bandwidth simply wouldn’t allow it.
What Needs to Happen Locally
A robotic arm can’t wait for a round trip to the cloud to determine whether it’s about to collide with something. Some control loops require sub-10-millisecond response times. Models take time to run, and network comms add to that latency. A delayed decision is, at best, a missed action and, at worst, a catastrophic system error.
Then there are sensor modalities and sensor and system specs. You need to understand how to interpret the data based on how it was captured, what sensors were used and how they were installed, where the machine was, and what it was doing. You also need to know how to align and enrich different data sources and in what order. Reconstructing this context later can be expensive or almost impossible. These systems need to combine the relevant data so they can act with the right context. Models are rigid systems: without the right context, garbage in, garbage out.
There’s also the network connectivity. The system must continue working reliably when the connection drops. Once the connection returns, data syncing must fit within the available bandwidth, some of which must always be reserved for critical operations. Some deployments also restrict where local data can be sent or processed, so you need to decide what can leave the device and what must run on it.
Keeping Data for Training and Testing
For systems like Waymo, sensor inputs are processed and driving decisions are made in real time. They also capture edge cases to later teach models new behaviors, such as responding to new road conditions or traffic laws. To investigate a failure or test a model change against a recorded scenario, they need time-aligned sensor inputs and system metrics. They must capture that information while the systems are running, but capturing everything isn’t practical. As part of edge processing, they need to selectively record the streams and metrics most useful for later analysis.
What gets recorded on the device determines what’s available later. If you omit important metadata, you may not be able to reconstruct the scenario for training or testing. Recording choices should therefore be made in parallel with decisions about what runs on the device during system design.
Working With the Hardware You Don’t Have Physical Access To
The models must share limited GPU, CPU, RAM, and storage with other software onboard. On battery-operated devices, there’s a power budget, too. You’re working with hardware already in the field; you can’t just scale horizontally as you would in the cloud. E.g., not all models are created equal; systems may need an orchestration layer to select the right model(s) and hierarchy for each task, optimizing compute and latency.
There’s again the data to manage: what to keep in memory, what to save to local storage, what to filter out at the source, and what to delete. You also need to decide what to transfer and when, based on available bandwidth and the device’s workload.
Software updates must always be compatible with the system, kernel versions, and runtime dependencies (CUDA is a prime example). Across a fleet, rollouts need to be coordinated. A failed update can leave a device unable to boot, requiring someone to recover it on site.
After a hardware restart, the system needs to resume operation on its own. It also needs a way to recover from failures without physical access. Data uploads and software updates must not interfere with key operations. If you didn’t build these from the start, every new device you ship adds a massive maintenance cost.
Close the loop
A production-grade system must support the full sensor-to-action loop:
- Access - capture sensor data and its context at the source
- Understand - process and reason about the data and the environment in real time locally
- Act - respond within the system’s timing requirements with an understanding of the system and the world around it
For physical systems, edge infra is part of the foundation. The cloud can handle training, testing, and high-level objectives while those systems handle time-sensitive decisions, data capture, and recovery. Design that split from the start.