I See What You Want:Decoding Hierarchical Operator Intent
from Visual Teleoperation History

Xiangyi Jia¹Kaixian Qu¹Shengzhi Wang^1,2Chenyu Yang¹Robert Katzschmann¹Marco Hutter¹

¹ ETH Zurich² Chinese University of Hong Kong

Abstract

Vision-language models (VLMs) have become a powerful foundation for robotic scene understanding, task planning, and language-conditioned control. A natural question follows: can these models also decode what a human operator intends while teleoperating a robot, without any explicit instruction? We formulate this as hierarchical intent prediction, jointly inferring the ongoing atomic action and the broader mission it serves from a short visual teleoperation history.

To study this, we introduce VI AMO (Vision Intention Action and Mission Ongoing), a dataset of 6,600 episodes from eleven robot sources with 13,653 temporally aligned action annotations and 9,702 mission annotations. We find that large proprietary models struggle with this problem: on test episodes, GPT-4o reaches 0.570 and 0.476 Sentence-BERT similarity for action and mission, and GPT-5.6 Luna 0.588 and 0.562. Fine-tuning a VLM on VI AMO yields our Vision-Intention Model (VIM), which raises both to 0.824 and still reaches 0.736 and 0.741 on more than one third of test actions and missions whose labels never appear in training.

Finally, as a downstream application, VIM’s predicted mission lets a vision-language-action (VLA) policy take over from the operator mid-task in a real-world environment unseen in training, reducing human teleoperation time across five real-world tasks by 34.6% relative to GPT-4o and 39.8% relative to Luna, with the latency reduced by approximately 70% on average.

Video

VI AMO Dataset

6,600 episodes

11 robot sources

13,653 actions

9,702 missions

Dataset composition

View source counts

VI AMO source composition
Source	Episodes	Actions	Missions
Bridge V2	3,091	4,950	3,445
DROID	2,041	6,372	4,452
Fractal / RT-1	579	656	579
TACO Play	403	403	403
CMU Play Fusion	224	725	448
RH20T	95	224	143
Jaco Play	68	68	68
UT Austin MUTEX	43	88	71
Berkeley FANUC	27	48	32
FastUMI	26	100	52
Stanford HYDRA	3	19	9

Animated montage of VI AMO demonstrations from eleven robot sources.

OM: Pour the juice into the cup and return the bottle to the coaster.

OA1 · Pick up the bottle

OA2 · Pour the juice into the cup

OA3 · Return the bottle to the coaster

Annotation pipeline, dataset, and VIM training. — **Annotation, dataset, and training.** Robot demonstrations are annotated with temporally aligned actions and missions. VIM combines visual history with completed-action context to generate the current action and its broader goal.

Intent Prediction Results

Prediction from partial observations

Across nine observation cutoffs spanning 0–95% of an action, VIM achieves higher action and mission similarity than the six baselines under both SBERT and SigLIP. Additional visual history generally improves predictions.

Four plots compare seven models across action progress: ongoing action and mission similarity measured with SBERT and SigLIP. VIM leads across the observation cutoffs. — **Intent prediction across action progress.** Curves show mean similarity on the VI AMO test set. All models use the same evaluation examples and observation cutoffs; higher is better.

Generalization to unseen labels

On test actions for which both action and mission labels are absent from training, VIM reaches 0.736 action SBERT and 0.741 mission SBERT. This evaluation measures unseen label strings within the dataset’s source mixture.