Real Nintendo Switch · no emulator · no ROM

An AI that beat Pokémon FireRed on a real Switch.

It sees the game through a capture card and plays with an emulated Bluetooth controller. A language model plans; deterministic code walks, battles and reads every pixel. Below, the whole campaign replayed from its own diary.

Watch the replay Source code

Frames captured from the Switch during the campaign — Frames captured from the console during the run

Interactive replay

Six days of play, event by event

Every point is a real event from the agent's log: where it stood, what it decided, the team it carried. Drag the timeline or press play; the markers jump to badges, story milestones and captures. The agent's own notes are shown as it wrote them, in Portuguese.

——

—

badgestory milestonecapture

Event log

Loading the campaign…

Team

Badges0/8

Model's latest decision

—

Objective

—

How it works

Pixels in, button presses out

The only sensor is the 240×160 picture and the only output is a controller. Everything else is reconstructed from the game's own decompiled source.

Perception

The frame is matched against every map rendered from the pret/pokefirered decomp to get the exact tile; text is read with the game's own font; species are recognised by sprite.

fireredia.percepcao

Skills

Path finding across the whole world (doors, ledges, Surf, Strength, ice) runs closed-loop, checked frame by frame. Wild and trainer battles, menus and the PC run without the model.

fireredia.habilidades

Agent

A strategist model writes the plan; an executor plays it through ~60 tools. Memory is an append-only log, so every episode starts small and nothing is forgotten.

fireredia.agente

Switch→HDMI · OBS→perception→skills ⇄ agent→NXBT in WSL→Bluetooth→Switch

Localisation: the Switch frame (left) and the window found in the map rendered from the decomp; the red square is the player's tile.

Dashboard — The live dashboard: screen, objective and plan, alerts, journey progress, team, bag, Pokédex, current map with the agent's trail and the full event log.

Post-game

Seven legendaries, all shiny

After the Champion, scripted missions and arbitrary code execution (code typed into PC box names through the Mail glitch) caught and recoloured them. Every trigger saves first and checks the boxes on screen.

Deoxys plan — The Deoxys triangle: 11 touches, each within the game's step budget (sDeoxysStepCaps). Planned by search, solved on the first try.

Ice plan — Icefall Cave: four slides on the ice floor to reach the HM07 ladder, found by a BFS over stopping points.

Shiny Deoxys, star checked on the summary page

The ACE code, written as PC box names 1–11

Engineering notes

What real hardware teaches

Lost inputs are normal

Bluetooth drops button presses, mostly during animations. Every multi-step action checks the screen before it commits: battle cursor, PC panel, keyboard page and cursor.

The map data isn't the whole truth

Scripts swap metatiles and flags hide NPCs. The navigator models script-opened doors and takes the list of NPCs the story already moved.

Reproduce the engine, not intuition

Leaving Surf only works onto elevation 3 (CanStopSurfing); ice slides until it hits something; the Deoxys triangle has a step budget. Each rule became code and a test.

Tested offline

36 tests run without a console: planning on synthetic grids, navigation on the real decomp maps, perception on frames captured from the Switch. CI runs them on Windows.