Play Pokémon Blue with an agent
The Gemini Plays Pokémon agent reads game RAM and screenshots, turns actions into button presses, and uses pathfinder and boulder-puzzle sub-agents.
What they did
An independent developer, Joel Zhang, built a harness that gives Gemini 2.5 Pro a subset of Game Boy RAM as text, overlaid on a screenshot. Gemini picks actions that become button presses. The harness summarizes history every 100 turns and runs a critic instance every 25 turns. It also provides pathfinder and boulder-puzzle sub-agents, which are prompted Gemini instances.
How it went
The agent beat Pokémon Blue in 813 hours on run 1, then in 406.5 hours on a fully autonomous second run. It looped on long contexts, hallucinated items from other games, and handled raw pixels poorly.
Worth knowing
The model struggled to read raw screen pixels, so the RAM-to-text translation was needed. Behavior also degraded once context grew well past 100k tokens.
Labels
Play Pokémon Blue with an agent
The agent beat Pokémon Blue in 813 hours on run 1, then in 406.5 hours on a fully autonomous second run.
The agent beat Pokémon Blue in 813 hours on run 1, then in 406.5 hours on a fully autonomous second run.
Gemini picks actions that become button presses.
An independent developer, Joel Zhang, built a harness that gives Gemini 2.5 Pro a subset of Game Boy RAM as text, overlaid on a screenshot.
The “Material” label failed its quality check and is not used for selection.
Details
- Agents
- Gemini
- Source
- other
- Source date
- undated
- Accounts
- 1
- In registry since
- Oct 12, 2026