Teleoperation, simulation or human video? Three sources of robot training data

Robots learn from three main sources of data: teleoperation, where a person controls the robot to demonstrate a task; simulation, where the robot practises in software; and human video, where people do the task on camera. Each gives something the others lack, so teams usually combine them.

This article compares the three, explains what each takes to produce, and shows how to decide on a mix. For a closer look at human video, see Egocentric video for robot learning.

What is teleoperation data, and what does it give?

Teleoperation data is data recorded while a person controls a real robot to perform a task. It gives robot-native data: the robot's own camera images and sensor readings together with the commands the operator sent to that robot.

Robot-native data is the strength of teleoperation. A policy can be trained on it directly, because observation and action come from the same machine. The authors of HumanScale (2026), a preprint, describe teleoperated real-robot trajectories as the dominant pretraining source for embodied foundation models, because of their precise action supervision and their match to the robot's body.

The weakness of teleoperation is collection effort. A person has to operate the robot for every demonstration, in real time, in a place where the robot can work. The HumanScale authors list high collection cost, difficult acquisition and low behavioural and environmental diversity as the limits of teleoperated data.

Large collections show what the effort looks like:

• DROID (2024) reports 76k demonstration trajectories, or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia and Europe over 12 months. According to the DROID paper's full text, its collectors controlled a Franka Emika Panda arm with the controllers of a Meta Quest 2 headset.
• Open X-Embodiment (2023) assembled data from 22 different robots, collected through a collaboration between 21 institutions, covering 527 skills.

Teleoperation data is also tied to one embodiment. Demonstrations on one arm do not transfer automatically to a robot with a different body, although Open X-Embodiment reports that training on data from many robots showed positive transfer and improved the capabilities of several of them.

What does simulation give, and where does it fall short?

Simulation gives cheap scale and exact labels for the simulated world: a simulated robot can practise many times, in parallel, and the simulator knows the exact state of every object it models. Its weakness is the sim-to-real gap. The sim-to-real gap is the difference between how a robot behaves in simulation and on real hardware.

In simulation, every contact, position and success can be measured, which makes it a natural place for reinforcement learning. Tasks can be reset quickly, varied freely, run in parallel and repeated without wearing out hardware. The labels are exact for the simulated scene, but only as faithful to reality as the simulation itself.

Simulated images, physics, friction and objects never match reality exactly. A policy that relies on details that differ, such as the exact look of a surface or how a soft object deforms, can fail when it meets the real world.

Domain randomisation is a technique that varies a simulation so widely, for example in textures, lighting and camera position, that the real world may look like one more variation. Tobin et al. (2017) describe bridging this "reality gap" as a way to improve the availability of data for robotics, and report training an object detector only on simulated images with random, non-realistic textures that worked in the real world. Their result is for object detection, not for a full control policy.

What does human video give, and what is missing?

Human video gives variety and scale: people already do the work, in real places, with real objects, and a camera can record it. What is missing is the robot, because the video shows human hands, not robot commands.

The embodiment gap is the difference between the body that produced the training data and the robot that has to act, for example a human hand and a two-finger gripper. Narrowing that gap, for example by mapping hand motion onto a gripper, is a field of its own and is covered in Egocentric video for robot learning, not here.

Egocentric video is first-person video recorded from the point of view of the person doing the task, usually with a head-mounted camera, so it keeps the hands and objects in view. Published work uses egocentric video in two main ways:

• Visual pretraining. R3M (Nair et al., 2022) pretrained a visual representation on the Ego4D human video dataset and reports that it let a Franka Emika Panda arm learn a range of manipulation tasks in a real, cluttered apartment from just 20 demonstrations.
• Pretraining larger models. The HumanScale authors report that, with the same amount of pretraining data and a careful filtering and labelling pipeline, models pretrained on egocentric video reached lower validation loss on real-robot action prediction than models pretrained on teleoperated robot data, mostly on tasks not seen in post-training; on seen tasks the two were close. Validation loss is an offline measure, not a test on a robot; according to the full text, the paper's real-robot trials compare egocentric pretraining with no pretraining, not with robot-data pretraining. Their recipe is to pretrain on human video and adapt with a small amount of labelled robot data.

Human video has costs of its own: people must consent to being recorded, bystanders must be protected, and the footage must be filtered and labelled. Consent and privacy in real-world video datasets covers the privacy side.

How do the three sources compare?

The three sources trade robot-native actions against scale and variety. No source wins every row of the comparison.

Qualitative comparison; effort varies by task and setup.

What does each source take to produce?

Teleoperation takes robot time and operators, simulation takes engineering up front, and human video takes processing after recording. The kind of effort is what limits how much of each source a team can get.

Teleoperation takes a robot, an operator and a place to work, for every hour of data. Collection scales with the number of robot setups and trained operators, and each setup produces data for its own robot.

Simulation takes engineering up front: scenes, object models, physics settings and the task logic. Once built, a scene can produce data almost without limit, but each new kind of object or material needs new modelling, and checking that the simulation is close enough to reality needs real tests. Soft or deformable materials, such as cloth or food, are generally harder to model than rigid objects.

Human video takes people who do the work anyway, cameras, consent and processing. Recording itself adds little to the work, but every recording has to be checked, cut into episodes, labelled and cleared of personal data before it is useful. The processing can easily be the larger part of the effort.

What labels does each source come with?

Teleoperation comes with recorded commands, simulation with exact labels for everything it models, and human video with almost no labels until someone adds them. The gap between what a source has and what the model needs decides how much extra work is required.

Qualitative comparison.

Human video needs the most labelling, and simulation the least.

How do you decide what data a robot needs?

The data a robot needs follows from the properties of its task: what it must learn that it cannot learn from cheaper data. The table maps common task properties to a main source and to the robot data that is still needed.

Qualitative guide; every robot and task needs its own tests.

Video shows motion, not force, so force-heavy tasks lean on robot data, ideally with force or torque readings: teleoperation with motion controllers records the commanded poses, and forces only if the robot has force-torque sensing. Variety across real places is where human video is the cheapest source.

How do teams combine the three sources?

Teams combine the sources by using broad, cheaper data to learn general skills and a smaller amount of robot data to connect those skills to a particular robot. Two training strategies are common.

• Pretrain, then adapt. The model, or part of it, is first trained on the broad data, then adapted with robot data. HumanScale post-trains the pretrained model on 1,500 robot demonstrations, 100 for each of 15 tasks; R3M keeps its pretrained visual encoder frozen and trains a policy on top of it from robot demonstrations.
• Co-training. The model is trained on a mixture of sources at the same time, with each batch drawing from several datasets. Open X-Embodiment trained on data pooled from many robots and reports positive transfer, so one robot's data can help another.

Three combinations of sources appear in published work:

1. Human video, then robot data. Human video teaches what to look at and how tasks unfold; robot demonstrations teach the robot's own actions.
2. Data from many robots. Pooling robot data across embodiments, as in Open X-Embodiment, can help other robots, especially those with little data of their own; in the paper's full text, robots with large datasets gained only with a much larger model.
3. Simulation, then real hardware. Domain randomisation aims to make what is learnt in simulation carry over to real robots. Testing on real hardware remains necessary.

The order matters for cost. Human video and simulation can provide breadth before any robot is set up; teleoperation is then spent where only robot data can help.

What should a buyer ask about each source?

A buyer should ask each source the questions that expose its typical weakness: embodiment and specification for teleoperation, realism for simulation, and consent and labels for human video.

A teleoperation dataset is usable only if its specification matches the buyer's robot and training code. The specification should state:

• Robot and gripper: the model, and how close it is to the buyer's.
• Action representation: joint positions, end-effector poses or velocities, and in which frame of reference.
• Control frequency: how often commands were sent.
• Proprioception: which joint and gripper readings are recorded with each step.
• Force and feedback: whether force-torque readings were recorded, and what feedback the operator had.
• Calibration and time sync: camera calibration, and how cameras and robot readings share one clock.
• Episode format: the file format and how episodes are split.
• Success labels: whether each episode is marked as success or failure, and by whom.
• Failed and corrected episodes: whether they were kept, removed or marked.

For the other two sources:

• Simulation: Which simulator and physics settings? How was the simulation checked against real hardware? Which materials and objects are modelled, and which are not?
• Human video: Which workplaces and tasks? How was consent obtained and recorded, and how were bystanders removed? Which labels come with the video, and does the licence allow training and products?

For teams that choose human video, What makes robot training video useful? lists what to agree on before recording. To ask for first-person recordings of specific tasks, see Vencur Motion, which is in its pilot phase.