Robot demo, pilot or deployment: what evidence should you ask for?

A robot completing a task once is useful evidence of a capability. Deciding whether it can help a working site requires additional evidence about repetition, people and conditions. These are editorial distinctions, not formal certification categories.

Separate three stages

Demonstration

A task has been shown under stated conditions. Ask what was selected, edited or prepared, and whether a human controlled or assisted the robot.

Pilot

The system is being tried in a relevant setting. Ask how long it ran, which tasks counted as success and what the support team had to do.

Deployment

The system is used for an operational job. Ask about sustained useful output, interruptions, maintenance and whether the customer is willing to describe the result.

Ask for a task, a denominator and conditions

“It picks objects” is less useful than a description of the objects, allowed variation, attempted cycles, successful completions and interventions. Compare like-for-like conditions. A lab trial with prepared objects and an edited warehouse clip cannot be treated as equivalent reliability evidence.

Separate robot time from useful work

Ask whether reported hours include charging, waiting, setup or human recovery. Identify who maps the space, replenishes materials, clears failures and services the machine. Obtain the assumptions behind a claimed payback rather than accepting a purchase price as total cost.

Two examples of better questions

The ALBATROSS report describes a 150-metre drop test and about three hours of reservoir operation. Those trials support more than a concept illustration, while open-sea operation remains a different test. Ask which conditions were demonstrated and which are still planned.

Agility cofounder Jonathan Hurst’s Digit account explains the tension between elegant movement and a commercially useful warehouse job. It is valuable first-party engineering perspective; it does not independently verify every product claim. Ask the customer what changed in its actual workflow.

A record to request

• Exact task, input range and environment.
• Trials attempted, completions and human interventions.
• Test duration, useful throughput and downtime.
• Operator roles, setup, maintenance and charging.
• Evidence source, publication date and commercial relationship.
• Limitations and the next test that could change the conclusion.

NIST’s work on performance metrics for industrial robotic systems provides further context on measuring robot capabilities. This guide is an editorial checklist, not a safety assessment or purchase certification.

NIST robotic performance metrics · Digit founder’s account · ALBATROSS reporting