A robot completing a task is a capability signal. A robot completing the right work repeatedly, with a known amount of help, is a deployment proposition. The difference is not how impressive the video looks. It is what happens outside the successful attempt.
That distinction is becoming more important as robot foundation models move into customer environments and evaluation moves beyond a single success score. A useful assessment asks four questions: what work was attempted, how often it succeeded, who intervened, and what happened after something went wrong.
Start with the denominator
In February 2026, Physical Intelligence published partner accounts of its pi-0.6 models folding laundry and packaging customer orders. Ultra's section labels a continuous full-shift packaging video as 96.4% autonomous and explicitly describes a human-in-the-loop intervention system. Weave separately reports interventions per laundry load and autonomy as a share of total time.1
These are developer-and-partner reports, not an independent BIGFOOT field test. Their useful feature is that assistance is visible in the account rather than erased from the story. An autonomy percentage is not automatically an order-success percentage, an uptime figure or a labor-saving percentage. Each needs its own denominator and measurement procedure.
Before comparing two systems, establish whether a number describes time, attempts, items, accepted outputs or completed shifts. A highly autonomous system can still need frequent brief interventions. A high completion rate can include substantial human recovery. Neither observation makes a system worthless; each changes the deployment economics.
The task mix is part of the result
RoboArena offers a different kind of evidence. Its 2025 conference paper describes more than 600 pairwise real-robot evaluation episodes across seven policies and seven academic institutions. Evaluators chose varied tasks and environments, then performed double-blind policy comparisons on the DROID platform. Aggregated preferences produced a ranking.2
A distributed comparison helps test whether a policy's apparent strength survives changes in setting. It does not establish a factory's output rate or the probability that a particular customer order will be packed correctly. A preference ranking and an absolute success rate answer different questions.
The commercial implication is straightforward: a task list should accompany a headline metric. Rigid boxes, deformable clothing, reflective packaging and cluttered work areas should not disappear into one average. A robot can be a strong fit for a narrow workload without being a general solution for everything that resembles it.
Recovery deserves its own test
A September 2026 preprint, RoboRecover, examines what happens after execution has already gone off course. Its 2,000 scenarios span the RoboTwin and LIBERO simulation environments, with fixed 800/200 training/test splits on each. Rather than starting only from a clean reset, it reconstructs intermediate deviation states and asks policies to continue the original task.3
The authors report that one policy, UnifoLM, achieved 98.83% initial-state success on LIBERO but 48.00% recovery. Pi-0.5 had lower initial-state success, 94.17%, yet higher recovery, 64.40%. These are reported benchmark results, not physical deployment rates. The contrast shows why success from a prepared starting point cannot stand in for recovery evidence.3
A dropped corner of fabric, a displaced package or a missed grasp changes the next decision. A credible evaluation needs to show whether the robot detects the change, retries safely, completes the task or requests help. Repeating a clean demonstration does not answer that question.
Make human assistance legible
Teleoperation means a person is controlling some of the robot's actions. Supervision, approval and physical resets are other kinds of assistance. They should be named separately rather than hidden inside a broad autonomy claim. Human help may be a sensible part of the product; its frequency, duration and limits determine how the service can scale.
For example, ten short remote corrections per shift and one long on-site recovery may produce similar intervention time but very different staffing requirements. A robot that calls for help appropriately can be more useful than one that continues confidently with an incorrect result. These are evaluation considerations, not measured comparisons between the systems cited here.
Five numbers worth asking for
The following is a practical evaluation checklist, not a certification standard. Each figure should include a defined task mix, observation period and counting rule.
- Unassisted completion: accepted outputs divided by all attempted tasks, with retries and failures retained.
- Assistance burden: interventions per task or hour, plus their duration and whether help was remote or on-site.
- Recovery: how often the system completes a task after a deviation, and how often a person must reset it.
- Useful throughput: accepted outputs per elapsed hour, including recovery and ordinary waiting rather than only motion time.
- Repeatability: results across repeated days, operators, object variations and relevant environmental conditions, with uncertainty rather than a best run alone.
A continuous video with timestamps and an intervention log is more informative than a highlight reel. A prospective trial against a representative workload is more useful still. Safety and site suitability require their own assessment; impressive task performance does not establish either.
The practical conclusion
The strongest robotics story is not "no humans ever." It is a clear account of the work the machine can perform, the conditions it needs and the assistance it still requires. Demonstrations deserve attention. Deployment decisions deserve evidence that includes the ordinary failures between the impressive moments.
References
- Physical Intelligence, with partner-authored accounts from Weave and Ultra (February 24, 2026). The Physical Intelligence Layer. Developer-reported customer deployments and assistance metrics; not independently reproduced here.
- Atreya, P., et al. (2025). RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. Conference on Robot Learning, PMLR 305, 336-364. Distributed real-robot, double-blind pairwise evaluation; a policy ranking is not a production acceptance rate.
- Li, Y., et al. (September 24, 2026). RoboRecover: Benchmarking Robot Policy Recovery under Execution Deviations. arXiv:2609.28952v1, preprint. Simulation recovery benchmark; no field reliability or safety certification inferred.