← All work

Zombie Walk Home

Hackathon · DubHacks '26, University of Washington · October 2026

Zombie Walk Home

A walking game that a projector throws onto a walking pad. Flowers come down the lawn; you step on each one as it reaches the line. An AI therapist coaches every walk.

Winner, MLH Best Use of Gemini API, DubHacks '26 (the Devpost page, opens in a new tab)

Scroll to walk

The walk, in five lines

  1. Warm-up Walk at a steady pace. The game learns your normal step.
  2. Round 1 of 3 Flowers come down two lanes, one per foot, on a beat.
  3. Round 2 of 3 The fourth flower is golden. Each round is a little faster.
  4. Round 3 of 3 The last round. Home is at the end of the lane.
  5. Home Home before sunrise.

The setup

What you are looking at

A drawing of the rig in a dark room: a projector on a chair throws a green lawn of two lanes onto a walking pad; a teenager walking on the pad steps on a grumpy orange flower at a white line while two more flowers come down the lanes; a camera on a stand looks down at the belt; a phone on a tripod films the walker's legs; a laptop on a table shows a cartoon coach
A drawing of the rig, made for this page with an image model. The photographs of the real one are further down.
  1. The projector Throws the game onto the pad and the floor in front of the walker's feet.
  2. The game Flowers come down two lanes, one per foot. Step on each one as it reaches the white line.
  3. The camera above Finds each shoe, so the game knows where every foot lands.
  4. The phone at the side Films the walk. Knee bend, stride, step time and left–right balance are measured from it.
  5. Maya The AI therapist, on Gemini: a short tip every 20 seconds, what went well between rounds, a review after the walk.
My part
The rig and the room, the game and the server that joins the parts, the Devpost and the demo
Team
Jakhangir Tynshimov, Rahul Vikram, Arturo, Tram-Anh Truong
Built with
Gemini, Backboard, Perceptron, MediaPipe Pose, YOLO-World, Phaser, FastAPI, Next.js, a projector and a walking pad
Event
DubHacks '26, University of Washington, October 17⁠–⁠18, 2026
  • 8

    jobs for Gemini

    As Maya, the AI therapist: coach, review, plan, notes, chat, reward and voice.

  • 3

    rounds a walk

    Flowers on a beat, alternating feet, each round a little faster.

  • 2

    cameras

    One above the belt finds the shoes. A phone at the side measures the walk.

  • 20 s

    between tips

    One short tip at a time, and only inside the therapist's limits.

Why

Home exercises that do not get done

Children and teens who are given movement exercises do most of them at home, and home programmes fall apart. Across 13 studies, 34% to 79.2% of parents of children with cerebral palsy did not keep up the home exercise programme (PLOS ONE, 2024). About 1 in 345 children in the United States has cerebral palsy (CDC), and 5% to 6% of school-aged children have developmental coordination disorder (CMAJ, 2015).

Games help when they are part of therapy: across 12 trials, active video games improved gross motor skills in children with developmental disabilities, and the authors ask for caution. What a game at home is missing is someone watching every step. We built an AI therapist to do that, and hid it inside a game.

How a session works

Seven steps, the last one is memory

  1. Set up The therapist or a parent picks the player, sets safe limits (pace, flowers, rounds, session length) and chooses which AI features may be used.
  2. Walk and play The player walks on the pad while the game is projected in front of their feet. Gus the zombie has to get home before sunrise; the player steps onto the flowers in time with the beat.
  3. See the feet A camera above the belt finds each shoe, so the game knows where every foot lands. Perceptron can double-check each step the game scores.
  4. Measure the walk A phone at the side films the walk. MediaPipe measures knee bend, stride, step time and left–right balance while they play.
  5. Coach live Gemini puts a short tip on screen every 20 seconds and, in the rest between rounds, says out loud what went well and one thing to fix.
  6. Review After the walk Gemini writes a review from the numbers: what went well, what to work on, suggested settings, and a note each for the player, the parent and the therapist.
  7. Remember Backboard keeps a memory for each player, so the next session starts with last time's focus.

Best Use of Gemini API

Maya, eight jobs

Gemini is Maya, the AI therapist, end to end. Seven of her eight jobs run on a small fast model that answers in about a second; the eighth is her voice.

  1. A cue during the walk

    One short line from the live numbers. Any change to the game is clamped to the therapist's limits.

  2. Analysis between rounds

    The world stops and Maya says what went well and the one thing to fix. Then 3-2-1, and the next round starts by itself.

  3. The session plan

    The greeting and today's focus, built from the player's past walks.

  4. The end-of-walk note

    One note for the player, one for the parent, one for the therapist.

  5. The gait review

    A structured review of the side camera's numbers. Every finding must cite a measured metric.

  6. A chat with photos

    The walk's numbers, its review and up to four annotated leg photos go in as images, so Maya answers about what she sees in your own steps.

  7. The reward pick

    After a completed walk, the reward from the parents' wishlist that fits the effort.

  8. The voice

    Every spoken line is Gemini text-to-speech, cached per line, with the player's name taken out.

Two rules shaped all eight. The therapist sets the limits and Maya cannot exceed them. And nothing depends on her: without a key every job falls back to fixed rules and the browser's own voice.

The rest of the stack

Each for one thing

  • Gemini
  • Backboard
  • Perceptron
  • MediaPipe Pose
  • YOLO-World
  • Phaser
  • FastAPI
  • Next.js
  • Tailscale Funnel
  • Python
  • TypeScript
Backboard
Memory across sessions: one assistant per player. After every saved walk the result is written to it, and the next plan, greeting and coach context read it back.
Perceptron
A second pair of eyes. After each flower, a few timestamped frames of the feet: did the named foot really land on the target? Feet and lower legs only, never a face, and only with consent.
MediaPipe Pose
The phone at the side: knee bend, stride, step time and left–right balance, a live skeleton for the dashboard, and annotated photos of notable steps.
YOLO-World
The camera above the belt: it finds each shoe, run locally, so the game knows which flower each foot landed on. A bright-blob fallback covers a dark room.
Phaser
The projected game: the scenes, the flowers, scoring at the hit line, the coach's lines and the end-of-walk screen.
FastAPI
The game server: it relays camera events over a WebSocket, stores players, limits and sessions, and runs every AI call with a rule-based fallback.
Next.js
The dashboard for the player, the parent and the therapist: live cameras, game controls, each walk with its review, Maya's chat, settings and consent.
Tailscale Funnel
The public link, so judges and phones outside our network could open the dashboard.

The rig

The hard part was physical

Every time we rebuilt the rig, five things had to line up again: the projector's four corners on the belt, the overhead camera's calibration to the same field, the shoe detector's sensitivity to the room's light, the phone at knee height with the whole body in frame, and the browser's permission to play sound.

The moment it paid for itself: the first time a projected flower splashed under a real foot on the walking pad.

  1. Walking in place lands every step in its own lane, so plain walking scored everything.

    A step counts only for a timed lift above the player's own normal step.

  2. One camera cannot measure a millimetre of toe clearance.

    Measure what it can, centimetre lifts and step timing, and say so.

  3. A projector of about 150 lumens disappears in a lit hall.

    High-contrast art on a white board, inside a cardboard booth lined black.

  4. Gemini's voice needs about four seconds to arrive, and a safety timer cut lines mid-sentence.

    The timer is sized to the line being spoken.

The weekend, in frames

The walking pad in a dark room, seen from its front end, with the game projected over it: a lawn in two lanes on the belt, a pink flower in one and a violet one in the other, and the four corner marks of the walking zone on the floor around it
Saturday night, in a dark room: the game on the pad, and the four corners of the walking zone being marked.
View the picture of the pad larger
The four of us standing on the DubHacks stage under purple light, lanyards on, two of us holding small navy pouches
The four of us on the DubHacks stage, Sunday afternoon.
View the picture of the team larger
Watch the demo YouTube (opens in a new tab)

The booth

Three cardboard walls and a roof, lined with black bags. A white board for the picture, a hole for the lens, a lower one for the phone.

What it does not claim

A tool for the therapist, not a replacement

It is not a medical device and it does not replace therapy. The wording in the app stays with coordination, balance, strength and general mobility; there are no disease names and no recovery claims in it. On the walking pad it is for teens and adults within the pad's rated age, with a person beside them, a low speed and a stop button.

The side camera's numbers and the annotated leg photos reach Gemini only with the family's consent, and the player's name is taken out of anything spoken. Perceptron sees pictures of feet and lower legs only. The server keeps no video.

My part

What I did

The rig and the room were mine: the projector in its cardboard booth, the walking pad, the camera above the belt and the phone at the side, and lining all of it up again every time we rebuilt it. So were the game on the floor and the server that joins the parts, the Devpost page and the demo video.

My teammates built the shoe detection and the side-camera gait analysis, the Perceptron checks, Gemini's voice and the review.