What makes robot training video useful? A capture and labelling checklist

Robot training video is useful when both hands and the objects stay in view, the camera is steady and well lit, each take covers a whole task, and the footage comes with time-stamped steps, task metadata and consent records. Most of that is decided before and during recording, not afterwards.

This checklist is written from the recording side. It covers camera placement, framing, light, takes, labels, metadata, consent and delivery, and ends with a table to use on the day. For why first-person video matters for robots, see Egocentric video for robot learning; for how it compares with teleoperation and simulation, see Teleoperation, simulation or human video?.

Where should the camera go?

The camera should sit where it sees the task the way the worker does, usually on the head or the chest, and stay in the same position across recordings. Egocentric video is first-person video recorded from the point of view of the person doing the task.

Public datasets show the head-mounted approach in practice: the EPIC-KITCHENS project lists a head-mounted camera for EPIC-KITCHENS-100 and Full HD recordings for its kitchen video. Whatever the choice, agree it with the buyer, note it in the metadata and keep it fixed, so the footage is consistent.

Three framing rules help:

• Wide enough for both hands. The field of view should cover both hands at arm's length, including the edges of the work surface.
• Pointed at the work. Adjust the angle so the work area sits in the middle of the frame when the worker looks at it naturally.
• Fixed for the whole take. Do not adjust the camera mid-task; if it slips, end the take and start a new one.

What must stay in view?

Both hands and the objects they handle must stay in view for most of the task. A model learns from what it can see: a grasp that happens outside the frame teaches nothing.

• Hands. Keep both hands visible, including at the start and end of each step. Hand keypoints are points that mark the joints and fingertips of a hand in an image, such as the 21 hand-knuckle coordinates that Google's MediaPipe hand landmark model detects. They can only be estimated for hands that are in the picture.
• Objects and tools. The object being handled and the tool in use should be recognisable, not just a blur at the edge.
• Containers and targets. Where the object goes matters as much as the grasp: the rack, the shelf, the bin.

A simple measurable check is the share of frames in which both hands are found. Running an off-the-shelf hand detector, such as the MediaPipe model set to detect two hands, over a test take gives that share, and the buyer can agree a minimum before recording starts.

How do light and motion affect the footage?

Light and motion decide whether hands and objects are sharp. Motion blur within a frame comes from the exposure time: in dim light the camera exposes longer, so fast hand movements smear. More light allows a shorter exposure; a higher frame rate shortens the longest possible exposure, but in dim light the camera then raises gain and the picture gets noisier.

• Light. Work in the normal light of the workplace, but avoid filming straight into windows or lamps. If a workplace is dim, ask whether more light is possible without changing the work.
• Motion. Quick head turns blur the whole frame. Ask the worker to work at their normal pace but to avoid unnecessary head movements while handling objects.
• Reflections and screens. Mirrors, windows and screens can show people or personal information; note them for removal.

What resolution and frame rate should you use?

Agree resolution and frame rate with the buyer before recording; there is no single correct value. The right choice depends on the model the buyer trains, the size of the objects and how fast the hands move.

Make the decision with a short test take. Record a minute of the real task, look at it frame by frame together with the buyer, and check that small objects are recognisable and fast movements are not smeared. Then fix the settings for the whole project and record them in the metadata.

Which camera settings should be agreed for robot learning?

Camera settings that keep time and geometry stable across a take decide whether footage can later be used for robot learning, not only whether it looks good. Agree these checks with the buyer before recording; which of them matter depends on the buyer's model.

• Constant frame rate. Confirm that the camera records at a fixed frame rate, so that frame numbers map to time stamps.
• Calibration. Record the camera's intrinsic parameters or a short calibration clip; they make hand positions estimated in 3D later more accurate.
• Locked exposure, focus and stabilisation. Settings that change during a take change the picture in ways a model may learn from, and electronic stabilisation crops and warps frames, so a calibration may no longer hold. Lock them, or agree which may change.
• Synchronised clocks. If other sensors record at the same time, such as a second camera or a motion sensor, keep their clocks in sync with the video.

Continuous takes or short clips?

Record continuous takes that cover a whole task, from the first step to the last. Short clips of the middle of a task lose what came before and after, and a model that learns the order of steps needs the whole sequence.

• One take per task attempt. Start before the first step and stop after the last.
• Keep the mistakes. If the worker drops something and picks it up, keep it: recovery is part of real work.
• Mark pauses. If the work is interrupted, note it, so the labeller does not mistake the pause for a step.

For a recording, an episode is one complete attempt by a person at one task, from the first step to the last. Delivering footage as episodes, one file or one marked segment per attempt, makes it easy to use.

How should steps be labelled?

Steps should be labelled as time-stamped action segments. Action segments are labelled stretches of video with a start time, an end time and a short description of the step; they tell a model what the person did and in which order.

A few rules keep labels consistent:

1. Agree the step list first. Write down the steps of the task and how fine-grained they should be, with the buyer, before labelling.
2. Use the same words every time. "Pick up plate" in one episode and "grab plate" in another make two labels for one step.
3. Mark start and end clearly. Give each step its own start and end event, agreed in advance: for example, "pick up plate" starts when the hand moves towards the plate and ends when the plate clears the rack, and "place plate in cupboard" ends when the plate is released.
4. Label failures. A step that fails, such as a dropped plate, should be marked as such, not left out.

Public datasets give a sense of how fine-grained labels can be. The EPIC-KITCHENS project lists 90K action segments for the 100 hours of EPIC-KITCHENS-100: about 900 segments per hour, or 15 per minute, roughly one every 4 seconds. A step list that fine suits models that learn individual motions; a buyer who needs only the order of larger steps, such as "unload lower rack", can choose a coarser list and save labelling time.

What task metadata should come with each recording?

Each recording should come with metadata that describes where, how and with what the task was done. Metadata recorded at capture time saves a great deal of work later.

• Place: the kind of workplace, such as a commercial kitchen or a workshop, without personal details.
• Task: the task name and the version of the step list.
• Tools and objects: the equipment used, and the main objects handled.
• Recording: date and a take ID; time of day only if the buyer needs it.
• Camera: device, lens or field of view, position, resolution, frame rate, calibration clip and any settings changed.
• Worker: a pseudonymous worker ID, with the key kept separately and handedness, without names.
• Conditions: lighting, interruptions and anything unusual about the take.
• Consent record: an identifier that links the recording to the consent of the person recorded.

How do consent and bystander removal fit into capture?

Consent and bystander removal belong in the capture plan from the start. Everyone recorded on purpose should give written consent before recording, and anyone else who appears should have their face and personal data removed before delivery.

• Before recording: explain the purpose and the use, obtain written consent, and agree the times and places of recording.
• During recording: keep other people out of frame where possible, and avoid filming screens, documents and name badges that show personal data.
• After recording: review every take, remove or blur the faces and personal data of bystanders, and record what was removed.

The Ego4D authors describe their collection as designed to uphold privacy and ethics standards with consenting participants and de-identification where relevant. Consent and privacy in real-world video datasets explains the GDPR side in more detail. This is general information, not legal advice.

What should a delivery contain?

A delivery should contain the video, the labels, the metadata and the consent references, organised by episode, in formats agreed with the buyer.

A manifest is a file that lists every episode in a delivery, with the files, labels and records that belong to it.

What does the checklist look like on the day?

The day-of checklist covers eleven checks, from camera position to the bystander review, each with a clear "done when". Go through it before each session and tick off each item.

To record your own hands-on work for robot learning, see Record your work for Vencur; to ask for recordings of specific tasks, see Vencur Motion, which is in its pilot phase.