EmbodiedMidtrain links VLMs to action
- Carnegie Mellon University and Bosch researchers posted EmbodiedMidtrain on April 21, describing a new training stage that adapts vision-language models for robot control. - The paper says a lightweight proximity estimator picks robot-relevant samples from large vision-language datasets, then improves results across three manipulation benchmarks. - The work targets a known data mismatch between web-trained models and robot trajectories. (arxiv.org)
Vision-language-action models are robots’ version of “see, read, do” systems: they look at a scene, parse an instruction, and output movements. Most still start from vision-language models trained on web data, not robot experience. (arxiv.org 1) (arxiv.org 2) A paper posted April 21 by Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, and Chenyan Xiong says that handoff leaves a gap between what the model has learned and what a robot needs to do. The authors call their fix EmbodiedMidtrain. (arxiv.org) The basic problem is distribution mismatch, a machine-learning term for training on one kind of data and deploying on another. In this case, web-scale vision-language data cover captioning, question answering, and documents, while robot data are manipulation trajectories grounded in physical interaction. (arxiv.org) EmbodiedMidtrain adds a middle step between those two worlds. The system uses frozen vision-language-model features and a lightweight learnable proximity estimator to find samples from a large vision-language pool that look most similar to robot data. (arxiv.org 1) (arxiv.org 2) The model is then mid-trained on that curated mix before the usual robot fine-tuning. The paper reports that this setup improved downstream performance across different vision-language-model backbones on three robot manipulation benchmarks. (arxiv.org) The authors say the gains showed up early in fine-tuning and widened as training continued, which they present as evidence of a stronger initialization. Their analysis also says the selector favored spatial reasoning examples over text-heavy tasks while keeping the source data diverse. (arxiv.org) That addresses a bottleneck that shows up repeatedly in the robotics literature: vision-language-action systems need far scarcer and costlier data than ordinary web models. A 2025 survey of the field counted 102 vision-language-action models and highlighted scalable pretraining and multimodal alignment as open problems. (arxiv.org) EmbodiedMidtrain does not claim to replace robot fine-tuning with internet data. It claims a better bridge into that stage, using selected vision-language examples to make the starting point more compatible with embodied control. (arxiv.org) The authors say they will release code, data, and models. For now, the paper’s main claim is narrower: if robots need models that can see and act, the training path may matter as much as the model itself. (arxiv.org)
