Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs
Abstract
- We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework designed for zero-shot, high-level physical task planning.
- Our approach integrates VLLMs into perception, control, and planning, enabling action sequence generation for novel environments from just an input image and a task description.
- Wonderful Team employs a multi-agent architecture to break down complex tasks, effectively handling longer sequences and ensuring reliability through self-correction.
- In real and simulated experiments, Wonderful Team demonstrates notable improvements over existing methods:
- +40% success rate on VimaBench compared to NLaP
- +30% improvement on standard tasks and +70% improvement on semantic reasoning tasks over Trajectory Generators
- These results highlight the growing potential of VLLMs in high-level robotic planning, reflecting their rapid advancement over the past year.
Method
We propose Wonderful Team, a zero-shot, single-model, multi-agent system for solving visual robotics tasks. Taking inspiration from recent advances in the multi-agent LLM literature, our system employs specialized agents to collaboratively manage different task aspects, from high-level planning to low-level execution, within a single integrated system. In particular, we develop a multi-agent LLM system wherein each agent is responsible for a separate component of task execution: including planning, object identification and location, action proposal, memory, and self-correction.
Results Overview
Execution Examples
[Real] Spatial Planning
[Real] Fruit Placement
“Place each fruit in the area that matches its color, if such an area exists.”
[Real] Price Ranking
“Based on the price tags and any discounts on the fruits, rank them from the most expensive to the cheapest and place them in the corresponding bowl.”
[Real] Superhero Companions
“Fruits and snacks of similar color make perfect companions. Distribute the unmatched items from the top left corner to the superheroes to help each of them have companion pairs.”
[Sim] Same Texture
“Put all objects with the same texture as {Object} into it”
[Sim] Same Shape
“Put all objects with the same profile as {Object} into it”
[Sim] Pick & Restore
“Put {Object_1} into {Object_2} then {Object_3}. Finally restore it into its original container”
Citation
If you find Wonderful Team useful in your research or applications, please consider citing it using the following BibTeX entry:
@misc{wang2024wonderfulteam,
title={Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs},
author={Zidan Wang and Rui Shen and Bradly Stadie},
year={2024},
eprint={2407.19094},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2407.19094},
}