Step 2: Let your agent see the map

The game server can send your agent a screenshot of the whole map, with every player and the chat on it. Your agent earns XP the first time it reaches each of the locations on the map, so being able to see them comes in handy. Pick a model that understands images, give your agent a tool to look at the screenshot, and let it find its own way.

1. Pick a model that can see

   For example, google/gemini-4-argon. Any model on the AI Gateway model list with image input works.

2. Ask your agent

   Give my agent in `backend/` a new `look` tool that fetches a screenshot of the game and hands the image back to the model. Switch the model to `google/gemini-4-argon`. Tell the agent that it earns XP the first time it reaches each location on the map, so it should look around and go visit them.

   Copy

   What your agent is going to implement

   1. Grab the screenshot

      One GET to the game server returns a PNG of the game right now.

      # On the game server. No auth headers needed.
      GET /game/controls/screenshot
      
      → image/png  # the map, every player, speech bubbles, and the chat

   2. A look tool

      The agent calls it whenever it wants to see. On Python, AI SDK for Python tools can return the image straight to the model.

      import ai
      import httpx
      
      # Any model with image input works. Check the Gateway model list for others.
      model = ai.get_model("google/gemini-4-argon")
      
      @ai.tool
      async def look() -> ai.messages.ContentOutput:
          """Take a screenshot of the game: the map, every player, and the chat."""
          async with httpx.AsyncClient() as http:
              png = (await http.get(f"{game_url}/game/controls/screenshot")).content
          # Tools can return images, not just text. The model sees the picture.
          return ai.content_output(
              "Here's the game right now.",
              ai.file_part(png, media_type="image/png"),
          )
      
      class GameAgent(ai.Agent):
          TOOLS = [spawn, move, look]  # same loop as before, now with eyes