First-Person Video: A New Gateway for Robot Data

First-Person Video Is Being Revalued

As of July 25, a recent round of discussion around LingBot-VLA 2.0 has pushed first-person human video back to the center of the data debate in robot learning. According to recently released information, this work combines first-person data with robot trajectories for training and attempts to determine whether videos of everyday human activities can improve the manipulation generalization of vision-language-action models, or VLAs.

This deserves attention not because robots can finally copy humans’ hand movements.

Quite the opposite: there is a clear embodiment gap between human hands and robotic arms. They differ in the number of joints, ranges of motion, and end effectors. Even for the same action, such as grasping a cup, the contact points and force application may be entirely different. Directly mapping hand trajectories in video to robotic-arm control commands is generally neither stable nor economical.

What first-person video may actually provide is an understanding of how the visual world changes before and after an action.

For example, when a person is about to open a drawer, the video will often show the following sequence:

1. The field of view shifts toward the drawer;
2. The handle moves into the center of the frame;
3. The hand approaches the handle, gradually increasing occlusion;
4. The drawer’s front panel begins to move;
5. The person’s gaze shifts toward the target object inside the drawer.

A robot may not need to reproduce the angle of the person’s wrist, but it can learn the task structure: first locate the handle, then establish contact, confirm that the drawer’s state has changed, and finally shift attention to the next stage.

This is more valuable than simply imitating an action trajectory and is also more likely to transfer across robot embodiments.

Robots Need More Than Just Action Data

Over the past few years, robot foundation models have faced a very practical data bottleneck: text and images can be obtained from the internet, but real robot trajectories must be collected using physical hardware.

Collecting a segment of robot data requires a robotic arm, cameras, teleoperation equipment, a suitable site, and human labor. If a grasp fails, operators must also deal with resetting the setup, collisions, and scattered objects. Different laboratories use different robot embodiments, camera positions, and control frequencies, making it difficult to merge data as readily as web text.

This is where first-person video becomes attractive. Action cameras, head-mounted devices, and smart glasses have already accumulated vast amounts of footage of human activities, including long-tail tasks such as cooking, organizing, repairing, assembling, and cleaning. Compared with third-person video captured by fixed cameras, first-person footage is generally closer to what a robot’s wrist-mounted or head-mounted camera sees: target objects appear larger, contact regions are clearer, and the object receiving the operator’s attention is more likely to appear in the center of the frame.

But a note of caution is necessary: a large volume of data does not necessarily provide effective control signals.

Human videos generally lack labels such as robot joint angles, gripper aperture, torque, and control frequency. Camera shake also entangles head movement, body movement, and hand actions. First-person video is better understood as a source of inexpensive but incomplete observational data than as standard trajectories that can be used directly for behavioral cloning.

The appropriate question, therefore, is not: Can human video replace robot data?

It is: Which parts of the problem can human video solve in place of robot data?

The answer is likely visual prediction and task priors, not low-level control.

What Really Transfers Is State Change and the Sequence of Attention

A typical VLA model must solve problems at three levels simultaneously:

• Perception layer: What objects are present in the scene, and what states are they in?
• Planning layer: Which stage has the current task reached, and what should receive attention next?
• Control layer: How far should the robotic arm move, and when should the gripper close?

Human first-person video is most likely to help with the first two layers. Its usefulness for the third depends heavily on subsequent alignment with robot trajectories.

1. Object Affordances

From large volumes of video, a model can observe that cup handles are generally best approached from the side, that drawer handles must first be contacted and then moved in a particular direction, and that fabric and rigid objects undergo clearly different state changes. Such knowledge is commonly referred to as affordance: the actions an object permits.

It does not provide precise action labels, but it can narrow the policy search space.

2. State Transitions Before and After Contact

The essence of robot manipulation is not making the end effector follow an elegant curve, but changing the state of the world. A bottle cap goes from closed to unscrewed, a block moves from the table into a box, and a towel goes from spread out to folded. Such changes are generally observable in first-person video.

If a model can predict the visual state at the next moment, or the relationships among objects several seconds later, it may develop a coarse-grained world model. Even if this capability does not immediately improve task success rates, that does not mean the pretraining was ineffective. The benefit may simply remain within the visual representation layer and fail to propagate successfully to the action head.

3. Task-Stage Structure

Long-horizon tasks are rarely a single continuous action. Making tea, for example, includes at least picking up a cup, getting the tea, adding water, waiting, and putting the kettle back. First-person video naturally records how attention shifts between objects, which can help models identify subtask boundaries.

It is important to note that camera orientation cannot simply be equated with human eye movements. Without an eye tracker, a first-person camera provides only weak proxy signals such as head orientation and the center of the field of view. Even so, it is generally closer to the operator’s current task focus than a distant, fixed camera.

A More Sensible Training Recipe

From an engineering perspective, first-person data should not be connected directly to the robot action head. A more robust approach is to use it for pretraining in observation space, then use real robot trajectories to complete control alignment.

The training process can be simplified into the following pseudocode:

for clip in ego_video:
    z_t = visual_encoder(clip.current_frames)
    loss_future = predict_future_latent(z_t, clip.future_frames)
    loss_state = predict_object_state(z_t)
    loss_phase = predict_task_phase(z_t)
    update_encoder(loss_future + loss_state + loss_phase)

for trajectory in robot_data:
    observation = align_robot_view(trajectory.images)
    action_chunk = policy_head(observation, trajectory.instruction)
    update_policy(action_chunk, trajectory.robot_actions)

There are three key points here.

First, the visual encoder needs to learn temporal continuity, not merely single-frame recognition. For a manipulation task, knowing that a drawer is present in the frame is far less useful than knowing that the drawer is being opened.

Second, the action head still requires robot data. First-person video can tell the model which object should be changed and what outcome to expect, but it cannot reliably specify how many degrees a particular joint should rotate.

Third, visual gains should ideally be distinguished explicitly from control gains. If pretraining improves next-state prediction accuracy without improving real-world task success, developers may need to examine not the quality of the video but the interface between the representation and the control policy.

This is also one of the most valuable points to emerge from recent discussions: do not look only at final success rates. End-to-end metrics are certainly important, but they combine errors in perception, planning, and control, making it difficult to determine where the information from first-person video is being lost.

Occlusion Is Both Noise and a Contact Signal

The most challenging issue with first-person video is that the hand often blocks the object at the most critical moment.

When grasping, pressing, rotating, or inserting, the operator’s hand often covers the contact region. From the perspective of pure visual reconstruction, this is missing data. From the perspective of task understanding, however, the occlusion itself may be a strong intent signal: a hand rapidly approaching and covering a bottle cap usually means that contact is imminent.

It is therefore unwise to discard all occluded frames. More appropriate approaches include:

• Using multi-frame context before and after contact to infer state changes during the occluded interval;
• Treating the hand region as an explicit mask so that the model can distinguish between “not visible” and “unchanged”;
• Jointly predicting hand-object contact probability, object state, and future frames;
• Incorporating depth, SLAM, or 3D hand-tracking results when available;
• Training with random temporal occlusion to prevent the model from mistaking the appearance of the hand for the task itself.

Evaluation is the truly difficult part. At present, it is hard to distinguish between two effects based solely on task success rates: did the model learn contact intent from occlusion, or did it merely learn to guess despite large amounts of missing visual information? Without tests specifically grouped by occlusion severity, contact events, and object visibility, these effects can easily become conflated.

Third-Person Controls Are Essential

First-person video may appear better suited to robots, but that does not mean it is inherently superior to third-person video.

If the first-person group uses 1,000 hours of video while the third-person group uses only 100 hours, the experiment may simply be measuring the difference in data volume. Similarly, recording quality, task variety, video duration, and action density can all affect the results.

A credible set of ablation experiments should include at least the following:

| Training configuration | Primary question addressed | | --- | --- | | Robot trajectories only | How strong is the baseline control capability? | | First-person pretraining + robot trajectories | Does first-person data provide gains? | | Equal amounts of third-person pretraining + robot trajectories | Do the gains come from the viewpoint or the volume of video? | | First-person video with shuffled temporal order | Does the model use state transitions? | | Video with occluded regions masked | Is hand occlusion informative or disruptive? | | Freeze the visual encoder and train only the action head | Can the visual representation transfer directly? | | Continue fine-tuning all modules | Does control alignment require reshaping the representation? |

Task success should not be the only metric. Evaluations should also examine at least:

• Prediction of the next state or future visual representations;
• Object-state recognition, such as open versus closed, empty versus full, and whether an object has been grasped;
• Contact-event detection;
• Subtask-stage recognition;
• Generalization to new objects, new backgrounds, and new camera positions;
• Collision rates, recovery capabilities, and completion times during real-world execution.

Special care must be taken to prevent data leakage. If the human videos and robot test sets contain highly similar scenes, objects, or task templates, apparent improvements in generalization may merely reflect visual memorization.

Developers Should Not Rush to Pile Up Video

If a team plans to experiment with first-person pretraining, the most important step is not to immediately collect tens of thousands of hours of video, but to first identify the capability gap they need to address.

If the model suffers from unstable gripper control, inaccurate force control, or calibration drift in the robotic arm, adding human video is unlikely to help. First-person data is more likely to be useful when the model can perform single-step grasps but cannot handle long-horizon tasks, unfamiliar objects, or verification of post-manipulation states.

In practice, teams can prioritize four things:

1. Increase action density: Ten hours of continuous walking footage is less useful than one hour densely packed with actions such as opening, closing, picking up, putting down, pouring, and inserting;
2. Preserve temporal context: Do not extract only keyframes; state changes before and after contact are the core information;
3. Standardize task semantics: Organize labels around actions, objects, and outcomes—for example, open—drawer—opened—rather than labeling only the drawer;
4. Establish a small-scale closed loop: First verify whether improvements in visual prediction transfer to dozens of real-world robot tasks, then decide whether to scale up the data.

Privacy and data licensing must not be overlooked either. First-person devices continuously capture faces, screens, domestic environments, and sensitive items, so their compliance costs may be far higher than those of ordinary object data. Scalability means more than inexpensive collection; it also means that the data can be used legally and reliably for training.

Assessment: It Is Better Viewed as Data for Visual World Models Than as Free Teleoperation

First-person video could become a new data source for robot models, but it is not an inexpensive substitute for robot trajectories.

Viewing it as free teleoperation data can easily lead development in the wrong direction. Human hand trajectories differ too greatly from the action spaces of robotic arms, and direct imitation introduces embodiment differences into the policy. Viewing it as a data source for visual world models and task structure is more consistent with current technical realities: models can learn object affordances, precursors to contact, state transitions, and shifts in attention from such video, while smaller volumes of higher-quality robot data can ground these capabilities in actual actions.

The significance of efforts such as LingBot-VLA 2.0 is not that a single experiment can prove the superiority of the first-person approach, but that they frame the problem more clearly: Did visual prediction improve? Did the control success rate improve? And was information lost in the encoder, cross-modal alignment, or the action head?

What robot learning truly needs is not another seemingly endless pool of data, but a pathway that explains how data is transformed into capabilities.

First-person video provides a promising entry point. Whether it can become infrastructure will be determined not by the number of video hours, but by rigorous equal-scale comparisons, layered evaluation, and the efficiency with which visual knowledge transfers to real-world control.

References

• Why first person video may matter for robot learning: A recent discussion of LingBot-VLA 2.0, visual-prediction ablations, third-person controls, and occlusion.
• 1,000 Hours of Human First-Person Data and Robot Task Data: An introduction to mixed training with first-person data and robot trajectories, as well as the goal of improving manipulation generalization.
• Human First-Person Data and Scaling Robotics: An overview of the evolution of first-person data for robot learning since Ego4D.
• Robots Watch Large-Scale Human First-Person Video: A discussion of attempts to use large-scale first-person video for visual prediction and robot world-model training.