The Core Metrics of Embodied AI

Episodes per success: the sample-efficiency metric

Episodes per success is the primary measure of how much experience a model requires to learn a specific manipulation or locomotion task. Training a policy from scratch in the real world can take tens of thousands of physical trials (episodes), which is wear-heavy and slow.

Embodied systems bypass this by pretraining on large visual datasets or using human priors to compress the representation space. For example, Tesla's FSD and humanoid projects leverage millions of miles of human driving and teleoperated manipulation data to seed policies before fine-tuning them on physical hardware.

Sim-to-real gap: the transfer metric

The sim-to-real gap measures the performance degradation when a policy trained in simulation is deployed on a physical robot. Since simulation is cheap and can run at 100x wall-clock speed, pretraining in sim is essential, but mismatch in friction, latencies, and sensor noise can cause sim-trained policies to fail instantly in the real world.

Closing the gap requires domain randomization (randomizing friction, mass, and lighting during simulation training) and system identification (precisely measuring physical robot parameters to calibrate the simulation). A low sim-to-real gap is what makes simulation work economically viable.

Reliability MTBF: the deployment-readiness metric

Mean Time Between Failures is the operational metric for commercial deployment. In robotics, a failure can mean a collision, a dropped object, or a joint lockup. While software systems can tolerate occasional crashes, physical failures incur capex damage and human risk.

For warehouse or manufacturing cobots, the threshold for commercial viability is typically 1,000+ hours MTBF. Achieving this requires moving beyond raw neural-network policies to hybrid control loops that wrap the model in deterministic safety barriers.

Latency budget: the control-loop metric

The latency budget is the maximum allowable time from sensor perception to motor actuator action. Unlike chatbots where a 500ms delay is acceptable, physical systems are governed by the physics of motion. A delay in braking or balancing results in a crash.

The budget scales down with speed and task precision: highway driving requires under 100ms end-to-end latency, arm manipulation requires under 10ms, and humanoid bipedal balancing requires under 1ms. This budget determines whether compute must sit on-device and how fast the inference model must run.

Throughput: the metric that couples success rate and speed

Success rate on its own is the most misleading number in robotics. A policy can score well by moving slowly enough to avoid every hard case, which is why the operators who actually buy robots measure throughput instead: units completed per hour, with failures counted. Throughput couples reliability and speed into one number, and it is the number that maps to a wage comparison.

Physical Intelligence reports its results this way. Across the training phases for a real cardboard-box workflow, pre-training to an SFT-like stage to reinforcement-learning post-training, the reinforcement-learning stage alone roughly doubled throughput. On espresso, a task requiring forceful portafilter insertion and precise timing, the same recipe reached over 90% success. The honest ceiling is stated in the same talk: the robot still makes mistakes, and it is still slower than a person.

This is the axis on which the model layer, not the hardware, is now moving. The box workflow, the arm, and the cameras did not change between those bars. The policy did.

Source: Chelsea Finn, Physical Intelligence, on Root Access (Y Combinator), 12 August 2026; transcript in `docs/articles/transcripts/Chelsea Finn - Next Decade in Robotics - Root Access.md`

Robot-hours: the denominator nobody can buy their way out of

Episodes-per-success only matters once you price an episode. In language models, an episode is a rollout in a data centre, so PPO and GRPO scale to millions or tens of millions of attempts by spending compute. In robotics the attempt happens on hardware, in real time, in the physical world.

Finn puts a figure on the translation: one million trajectories of a one-minute task, shorter than the espresso task, is roughly 700 robot-days. That is the constraint that makes physical AI structurally different from the model layer above it. You cannot buy 700 robot-days the way you buy a GPU cluster, which is why the sharpest work is aimed at the denominator rather than at capability.

Two attacks on it are worth naming. Human intervention terminates dead-end trajectories early, so hardware time is not spent on a robot trying to fold two stuck-together boxes. And a general-purpose value function, trained across all video experience to predict time-to-success, amortises value estimation across tasks rather than burning 10 to 50 rollouts per prompt to estimate it locally.

Source: Chelsea Finn, Physical Intelligence, on Root Access (Y Combinator), 12 August 2026

Autonomy horizon: how long it runs before a human steps in

MTBF measures how long the machine lasts. Autonomy horizon measures how long the policy lasts, and the two are not the same: a robot with perfect hardware reliability still needs a person the moment it loses track of what it has already done. This is the metric that decides whether a deployment needs a babysitter, which in turn decides whether it saves any labour at all.

Two numbers bound the current state. On a repetitive task, a latte-making policy ran 13 hours straight at useful reliability. On a non-repetitive one, kitchen cleaning across wiping, drying, putting away condiments and washing dishes, the horizon is 10 to 15 minutes of continuous autonomy.

The gap between those two figures is a memory problem, not a dexterity problem. Most state-of-the-art robot foundation models carry no context at all: they act on the current camera frame. Feeding video back in naively is prohibitive, since 10 seconds at 50Hz across four cameras at 256 tokens per frame is roughly half a million tokens in real time. The workaround is memory at two timescales, roughly 10 seconds of efficiently computed video memory for the motor loop, and natural-language summaries of completed sub-tasks for the minutes-to-hours span.

Source: Chelsea Finn, Physical Intelligence, on Root Access (Y Combinator), 12 August 2026

Generalist delta and cross-embodiment transfer

The last two metrics ask what the model can do without being trained for the specific job. Generalist delta is the performance gap between a single pre-trained policy used out of the box and a specialist fine-tuned for that one task. For most of robotics history the delta has been large and the specialist has won, which is why every deployment carried a fine-tuning bill.

That gap has now closed on the tasks measured. A single generalist model, π0.7, matches or outperforms the fine-tuned π0.6 specialists on coffee making, box building and garment folding, against both SFT and RL post-trained baselines. This is the BERT-to-GPT transition arriving in robotics: the point at which the pre-trained model is the product rather than the starting point.

Cross-embodiment transfer is the second measure. Garment folding trained on a standard bimanual platform transferred to a large industrial Byarm UR5E, a robot with different linkage lengths and joint configuration, with no folding data collected on it, approaching human teleoperation performance. Measured this way, data collected on one robot becomes an asset across a fleet, which is the only path by which the robot-hours denominator ever gets cheaper.

One ablation is worth carrying forward, because it inverts the usual data intuition. Without metadata prompting, adding the last tranche of low-quality data made performance worse. With it, the same data made performance better. Data quality is not a fixed property of a dataset; it is a function of whether the model is told what it is looking at.

Source: Chelsea Finn, Physical Intelligence, on Root Access (Y Combinator), 12 August 2026