Control robots with generated code and voice commands
Gemini detects objects and writes robot API code, and with 10 in-context demos an ALOHA robot packed boxes and folded a dress; voice commands became robot plans.
What they did
Gemini 2.5 was given a robot control API (detect object, get grasp pose, move/open/close gripper) and wrote Python plans zero-shot, such as putting a banana in a bowl. For ALOHA box packing and dress folding, 10 demonstrations of interleaved reasoning and actions were added to context. For voice, the same API calls were defined as Live API function-calling tools, with the camera feed streamed in.
How it went
Gemini produced two feasible banana plans, one of which re-planned around the right arm's reach limit. ALOHA packed boxes and folded a dress from the in-context demos. The post gives no success rates.
Worth knowing
The ALOHA setup is in an open-source Colab with example demonstrations. The Live API voice console is on GitHub, and Gemini Robotics-ER access requires a trusted tester waitlist.
Labels
ALOHA packed boxes and folded a dress from the in-context demos.
Gemini detects objects and writes robot API code, and with 10 in-context demos an ALOHA robot packed boxes and folded a dress; voice commands became robot plans.
The “Material” label failed its quality check and is not used for selection.
Details
- Agents
- Gemini
- Source
- other
- Source date
- undated
- Accounts
- 1
- In registry since
- Oct 12, 2026