Play a button-clicking game via a browser agent
A developer wired Gemini's Computer Use model to Playwright in Java and had it play a simple button-clicking game to verify click-coordinate scaling.
What they did
Guillaume Laforge built a Java loop around Gemini's computer-use preview model and Playwright's headless Chromium, with a 1000x1000 viewport. The model got a screenshot after every action and returned click actions with coordinates on a normalized 1000x1000 grid. His code scaled those to the viewport and clicked. He built a simple game where a red button moves after each click, and told the agent to click it 10 times and read the score.
How it went
The agent clicked the moving button and reported a final score of 10, which confirmed the click handling and coordinate scaling worked. The source mentions no failures in this test.
Worth knowing
Send a fresh screenshot to the model after every action, and wait about a second first so the page has finished rendering. Otherwise the model sees a blank or half-drawn page.
Labels
A developer wired Gemini's Computer Use model to Playwright in Java and had it play a simple button-clicking game to verify click-coordinate scaling.
He built a simple game where a red button moves after each click, and told the agent to click it 10 times and read the score.
The model got a screenshot after every action and returned click actions with coordinates on a normalized 1000x1000 grid.
Guillaume Laforge built a Java loop around Gemini's computer-use preview model and Playwright's headless Chromium, with a 1000x1000 viewport.
A developer wired Gemini's Computer Use model to Playwright in Java and had it play a simple button-clicking game to verify click-coordinate scaling.
The “Material” label failed its quality check and is not used for selection.
Details
- Agents
- Gemini
- Source
- other
- Source date
- Nov 3, 2025
- Accounts
- 2
- In registry since
- Oct 12, 2026