Octopus: Embodied Vision-Language Programmer from Environmental Feedback

Jingkang Yang* 1, Yuhao Dong* 2,6, Shuai Liu* 3,6, Bo Li* 1
Ziyue Wang †,1, Chencheng Jiang †,4, Haoran Tan †,3, Jiamu Kang †,2
Yuanhan Zhang 1, Kaiyang Zhou 5, Ziwei Liu 1,✉

1S-Lab, Nanyang Technological University    2Tsinghua University
3Beijing University of Posts and Telecommunications
4Xi'an Jiaotong University  5Hong Kong Baptist University
6Shanghai AI Laboratory

*Equal Contribution   Equal Engineering Contribution   Corresponding Author


Large vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. Furthermore, when seamlessly integrated into an embodied agent, it signifies a crucial stride towards the creation of autonomous and context-aware systems capable of formulating plans and executing commands with precision. In this paper, we introduce Octopus, an embodied VLM designed to 1) proficiently decipher an agent's visual and textual task objectives, 2) formulate intricate action sequences, and 3) generate executable code. Our design allows the agent to adeptly handle a wide spectrum of tasks, ranging from mundane daily chores in simulators to sophisticated interactions in complex video games. Octopus is trained by leveraging GPT-4 to control an explorative agent to generate training data, i.e., action blueprints and the corresponding executable code, within our experimental environment called OctoVerse. We also collect the feedback that allows the enhanced training scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a series of experiments, we illuminate Octopus's functionality and present compelling results, and the proposed RLEF turns out to refine the agent's decision-making. By open-sourcing our model architecture, simulator, and dataset, we aspire to ignite further innovation and foster collaborative applications within the broader embodied AI community. The codebase is released at https://github.com/dongyh20/Octopus.




          title={Octopus: Embodied Vision-Language Programmer from Environmental Feedback}, 
          author={Jingkang Yang and Yuhao Dong and Shuai Liu and Bo Li and Ziyue Wang and Chencheng Jiang and Haoran Tan and Jiamu Kang and Yuanhan Zhang and Kaiyang Zhou and Ziwei Liu},