Egocentric video for robot learning: a practical guide

Egocentric video is first-person video of a person doing a task, usually from a head-mounted camera, and robots can learn from it because it shows hands and objects in real work at a scale robot data rarely reaches. It does not show robot actions, so it complements robot data rather than replacing it.

This guide explains what egocentric video is, why robots learn from it, how published work turns it into actions, what public datasets offer and lack, and what to ask for when you need recordings of specific tasks. For a comparison with teleoperation and simulation, see Teleoperation, simulation or human video?.

What is egocentric video?

Egocentric video is video recorded from the point of view of the person doing an activity, usually with a camera worn on the head or the chest. It shows what the person sees: their hands, the objects they handle and the space directly in front of them.

The first-person point of view is what makes egocentric video useful for robots. A camera on a tripod sees a person from outside; a head-mounted camera sees the task close up, with hands and tools in the middle of the picture. That view is closer to what a camera on a robot's head sees than to what a fixed camera beside the robot sees; a wrist camera sees the work from even closer and lower.

Egocentric datasets often come with more than video. Ego4D, for example, accompanies portions of its video with audio, 3D meshes of the environment, eye gaze, stereo or synchronised videos from several cameras at the same event. Ego-Exo4D records the same skilled activities from first-person and third-person cameras at once.

Why do robots learn from human video?

Robots learn from human video because it is cheaper to collect at scale than robot data, and it shows far more variety: many places, many objects and many ways of handling them.

Robot data is collected on real robots, for example by people operating them remotely, which is slow and expensive, and each recording is made with one particular robot. The authors of HumanScale, a 2026 preprint discussed below, describe teleoperated real-robot trajectories as the dominant pretraining source, but limited in scale by high collection cost and by low behavioural and environmental diversity. Human video avoids most of that cost: people already do the work, and a camera records it.

Three properties make human video valuable:

• Scale. The Ego4D project page lists 3,670 hours of video from 923 participants in 74 locations across 9 countries.
• Variety. Real homes and workplaces contain the clutter, lighting and object variety that staged robot labs lack.
• Hand-object interaction. First-person video shows grasps, tool use and the order of steps in a task, which is what manipulation needs.

The catch is the embodiment gap, the difference between a human body and a robot's body: a human hand is not a robot gripper, and video contains no robot joint commands. Human video can show what to look at and what tends to happen next; turning it into robot actions takes extra work, described below.

How is egocentric video used for robot learning?

Published work uses egocentric video in two main ways: to pretrain visual representations and to pretrain larger models that are then adapted with robot data. The first has published support and has also been questioned; for the second, this guide relies on one recent preprint that compares it directly with robot-data pretraining. A third approach, learning actions without labels, is a variant of the second.

Visual pretraining

R3M (Nair et al., 2022) pretrained a visual representation on Ego4D and used it as a frozen perception module for robot policies. Across 12 simulated manipulation tasks, the authors report that it improved task success by over 20% compared with training from scratch, and by over 10% compared with other visual representations such as CLIP and MoCo. They also report that it let a Franka Emika Panda arm learn a range of manipulation tasks in a real, cluttered apartment from just 20 demonstrations. The percentage gains come from simulated tasks.

Not every study finds that egocentric data is the best choice. Dasari et al. (2023) compared pretraining datasets for visuo-motor representations and report that traditional vision datasets, such as ImageNet, Kinetics and 100 Days of Hands, are surprisingly competitive options, and that the image distribution of the pretraining data matters more than its size. They also report that common simulation benchmarks are not a reliable proxy for real-world performance, which is worth keeping in mind when reading gains measured in simulation. For a buyer, the lesson is that egocentric video earns its place when it shows the right images, the target tasks and places, not simply when there is a lot of it.

Pretraining embodied models

HumanScale (2026), a preprint, compared egocentric human video with teleoperated robot trajectories as pretraining data for the same model, with the same robot data afterwards. According to the full text, at 5,000 hours of pretraining the egocentric model reached a validation loss of 0.0204 on unseen tasks against 0.0254 for robot pretraining, about 20% lower, and was close on seen tasks (0.0067 against 0.0071). Validation loss is an offline measure.

On a real bimanual robot, the egocentric-pretrained model reached 92.5% success in distribution and 90% out of distribution, against 40% and 0% for the same model without embodied pretraining. The real-robot trials did not include a robot-pretrained model, so they show the value of pretraining, not a win over robot data. The authors call their results encouraging, but still preliminary, and their proposed recipe is to pretrain on egocentric video and then adapt with a small amount of labelled robot data.

How do you get robot actions out of human video?

Robot actions come out of human video only through estimation: the hands are tracked, placed in a fixed frame of reference and mapped onto the robot's gripper, or a model learns its own stand-in for actions. Each approach needs different data from the recording.

1. Track the hands. Hand keypoints are points that mark the joints and fingertips of a hand in an image. Google's MediaPipe hand landmark model, for example, detects 21 hand-knuckle coordinates in each detected hand. For robot actions, 3D positions are more useful than positions in the image.
2. Track the camera. A head-mounted camera moves with the head, so hand positions measured relative to the camera mix hand motion with head motion. The camera's own position over time separates the two. Meta's Project Aria post-processing services, for example, compute a closed-loop device trajectory that gives poses in a consistent frame of reference, and 21 hand landmarks in the device frame, in metres.
3. Map the hand to a gripper. Retargeting is the step that converts human hand motion into the robot's action space, such as a gripper's position and whether it is open or closed. The HumanScale authors estimate per-clip end-effector poses and gripper states from retargeted hand-pose signals and use them as pseudo-action labels, in the same action space as their robot data. They drew a 5,000-hour subset from the egocentric portion of the HumanNet dataset, selected for diversity in objects, skills and environments. The paper does not describe its 3D hand-pose method in detail.
4. Or learn actions without labels. LAPA (Ye et al., 2024) is a method for pretraining vision-language-action (VLA) models, models that take camera images and an instruction and output robot actions, without ground-truth robot action labels. It learns latent actions from videos that have no robot action labels, then fine-tunes on small-scale robot manipulation data to map latent actions to robot actions, so the robot still needs some data of its own.

Some newer datasets record the 3D hand data at capture time. EgoDex (Hoque et al., 2025) reports 829 hours of egocentric video with paired 3D hand and finger tracking data, recorded with Apple Vision Pro across 194 tabletop tasks.

The need for 3D hand and head data has a practical consequence for a recording project: how well hand paths can later be placed in 3D depends on what was recorded. Head pose can be estimated from the video afterwards, and a camera can often be calibrated later if its settings did not change, but such estimates are less accurate. Motion-sensor data and hardware time sync cannot be added afterwards. The latent-action route needs no calibration, head pose or motion-sensor data.

How do you turn raw egocentric video into training data?

Raw egocentric video becomes training data through a pipeline: check the footage, cut it into tasks and steps, label what matters, remove personal data and deliver it in an agreed format.

1. Check the footage. Remove takes where the hands leave the frame for long stretches, the camera points away from the task, or motion blur makes the objects unreadable.
2. Cut it into episodes. One episode is one attempt at one task, from start to finish, so that a model sees complete tasks rather than random clips.
3. Segment the steps. Action segments are labelled stretches of video with a start time, an end time and a description of the step, such as "00:04–00:07 pull rack" (illustrative). EPIC-KITCHENS-100 reports 90K actions in its 100 hours of kitchen video.
4. Add the labels the buyer needs. Objects, tools, hand keypoints, success or failure of each step, and task metadata such as the place and the equipment.
5. Remove personal data. Blur or remove the faces and other personal data of anyone who did not agree to be recorded, before the footage leaves the collector.
6. Deliver in an agreed format. Video files, label files and a manifest, a file that lists every episode with its parts.

Each step loses or adds information. Filtering removes footage, segmentation adds structure, and de-identification can hide details a model would have used, such as a label on a package or the display of a machine. Agree the trade-offs with the buyer before recording, not after.

What does a good egocentric recording look like?

A good egocentric recording keeps both hands and the objects in view, holds the camera steady, has enough light to avoid motion blur and covers the whole task from start to finish. Small choices at capture time decide how useful the footage is.

• Camera position. A head-mounted camera follows the worker's gaze; a chest-mounted camera is steadier but sees less of what the worker looks at. Agree the position with the buyer.
• Hands in frame. The hands should stay in the picture for most of the task, including at the edges of the work area.
• Light and motion. Fast hand movements blur when the exposure is long; more light allows a shorter exposure and sharper hands.
• Whole tasks. Continuous takes from the first step to the last are worth more than short clips of the middle.
• Sensor data. Camera calibration, head pose where the device records it, and synchronised clocks if there is more than one sensor.
• Metadata. Place, tools, objects and the task name, recorded at capture time, save a great deal of labelling later.

What makes robot training video useful? turns these points into a capture and labelling checklist, with resolution and frame rate agreed with the buyer rather than assumed.

When is egocentric video the wrong choice?

Egocentric video is the wrong choice when the robot needs exact forces or joint commands, when the task cannot be seen from the worker's head, or when the robot's body is so different from a human's that the motions do not carry over.

• Force and touch. Video shows where hands go, not how hard they press. Tasks that depend on force, such as fitting a tight part, need robot data or force sensors.
• Hidden work. If the decisive action happens inside a machine or behind the worker's body, a head camera does not see it.
• Very different robots. A mobile base with a single gripper is likely to learn less from two-handed human work than a two-armed robot would.
• Exact procedure. Where every motion must follow a fixed procedure, a few precise robot demonstrations may teach more than many varied human recordings.

In these cases, video can still help a robot recognise objects and steps, but teleoperation or simulation carries most of the learning.

What public egocentric datasets exist, and what are their limits?

The best-known public egocentric datasets are Ego4D, EPIC-KITCHENS-100 and Ego-Exo4D. They are large and well documented, but they mostly cover daily life, homes and skilled hobbies, and each comes with licence terms that must be read before use.

Three limits matter for robot learning:

1. Domains. Most footage shows homes, daily life and hobbies. Kitchens are well covered, and Ego4D includes hand-object activities such as knitting, carpentry and baking, but none of the three is built around repeated recordings of specific workplace tasks, such as warehouse picking or commercial cleaning. Newer manipulation datasets such as EgoDex focus on tabletop tasks rather than workplaces.
2. Licences. EPIC-KITCHENS is published under a non-commercial licence, with commercial licences available on request from its team. Ego4D and Ego-Exo4D require a signed licence agreement, and the official pages do not summarise its commercial terms, so the agreement itself must be read. None of this is legal advice.
3. Annotation. Labels were made for the datasets' own benchmarks, such as action recognition or episodic memory. They may not match the steps, objects or hand poses a robot team needs.

The papers also show what careful collection involves. The Ego4D authors describe a collection approach designed to uphold privacy and ethics standards, with consenting participants and de-identification where relevant. Consent and de-identification of that kind are the standard any new collection should meet.

What is missing from public datasets?

What is mostly missing is real workplace footage of specific tasks, recorded with consent and offered under terms that allow commercial training. A team that needs a robot to do one job in one kind of workplace rarely finds that job in public data.

• Real workplaces. Commercial kitchens, workshops, warehouses and cleaning work have tools, layouts and paces that homes do not.
• Specific tasks, repeated. Training needs many recordings of the same task in different places and by different people, not one example of many tasks.
• Commercial-use rights with consent. The people recorded must have agreed to the use, and the licence must allow training and products.
• The labels and sensor data you need. Step boundaries, object names, hand poses and calibration that match the robot's task.

Consent and privacy in real-world video datasets explains the privacy side of collecting such footage.

How might a project plan use egocentric video?

A project plan uses egocentric video for breadth and robot data for the robot's own actions, and measures whether the video earned its cost. A few dozen hours of task-specific video are an addition to a larger pretraining mix, not a pretraining corpus on their own: the HumanScale study scaled from 100 to 5,000 hours.

How does Vencur Motion approach egocentric video?

Vencur Motion is a pilot for first-person video of real hands-on work, recorded for robot learning. It starts with real workplaces and the written consent of everyone recorded on purpose.

• Status: Pilot. Contributors are paid to record their work; the contributor pilot runs in South Tyrol, Tyrol and Bavaria.
• Real workplaces. Kitchens, workshops, warehouses, cleaning and crafts, not staged scenes.
• Head-mounted camera. The person doing the work wears the camera on the head, so the video shows the task from their point of view.
• Consent and privacy. Everyone recorded on purpose gives written consent. If anyone else appears, their face and personal data are removed before delivery, and the work follows the GDPR.
• Annotation: Planned. Annotation of steps and actions is planned.

To ask for recordings of specific tasks, see Vencur Motion.

What should buyers ask for?

Buyers should ask for the tasks and places they need, proof of consent, removal of bystanders, the labels, sensor data and formats they need, and a licence that allows their use. A short sample before a larger order shows whether the footage fits.

Egocentric video is one source of robot training data among three, alongside teleoperation and simulation, and most teams combine them.

In this series

• Teleoperation, simulation or human video? Three sources of robot training data
• What makes robot training video useful? A capture and labelling checklist
• Consent and privacy in real-world video datasets: a GDPR primer