Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs

• We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework designed for zero-shot, high-level physical task planning.
• Our approach integrates VLLMs into perception, control, and planning, enabling action sequence generation for novel environments from just an input image and a task description.
• Wonderful Team employs a multi-agent architecture to break down complex tasks, effectively handling longer sequences and ensuring reliability through self-correction.
• In real and simulated experiments, Wonderful Team demonstrates notable improvements over existing methods:
  • +40% success rate on VimaBench compared to NLaP
  • +30% improvement on standard tasks and +70% improvement on semantic reasoning tasks over Trajectory Generators
• These results highlight the growing potential of VLLMs in high-level robotic planning, reflecting their rapid advancement over the past year.

We propose Wonderful Team, a zero-shot, single-model, multi-agent system for solving visual robotics tasks. Taking inspiration from recent advances in the multi-agent LLM literature, our system employs specialized agents to collaboratively manage different task aspects, from high-level planning to low-level execution, within a single integrated system. In particular, we develop a multi-agent LLM system wherein each agent is responsible for a separate component of task execution: including planning, object identification and location, action proposal, memory, and self-correction.