Teaching Claude to Catch 'Em All (or Repeatedly Walk Into Walls): A New take on ClaudePlaysPokémon

This presentation will showcase a novel Human-AI system where an AI agent, autonomously plays the classic game Pokémon Red. We will demonstrate how the AI analyzes the game’s visual state and internal memory (via the PyBoy emulator) to make real-time decisions, navigate the world, interact with characters, and engage in battles.
The core focus of the presentation is the collaborative gameplay dynamic between the human user and the AI agent. Attendees will see how users can observe the AI’s gameplay and provide high-level strategic suggestions through a simple text interface. These suggestions guide the AI’s decision-making process, allowing the human to act as a coach or strategist, influencing the AI’s goals and actions without needing direct control.
We will discuss the technical implementation, the challenges of translating complex game states for AI understanding, and the fascinating emergent behaviors observed during AI-driven gameplay. VibePlay serves as a compelling demonstration of interactive AI, exploring how humans and AI can collaborate in complex, dynamic environments like video games.