Hackathon · DubHacks '26, University of Washington · October 2026
Zombie Walk Home
A walking game that a projector throws onto a walking pad. Flowers come down the lawn; you step on each one as it reaches the line. An AI therapist coaches every walk.
Winner, MLH Best Use of Gemini API, DubHacks '26 (the Devpost page, opens in a new tab)Scroll to walk
The walk, in five lines
- Warm-up Walk at a steady pace. The game learns your normal step.
- Round 1 of 3 Flowers come down two lanes, one per foot, on a beat.
- Round 2 of 3 The fourth flower is golden. Each round is a little faster.
- Round 3 of 3 The last round. Home is at the end of the lane.
- Home Home before sunrise.
The setup
What you are looking at
- The projector Throws the game onto the pad and the floor in front of the walker's feet.
- The game Flowers come down two lanes, one per foot. Step on each one as it reaches the white line.
- The camera above Finds each shoe, so the game knows where every foot lands.
- The phone at the side Films the walk. Knee bend, stride, step time and left–right balance are measured from it.
- Maya The AI therapist, on Gemini: a short tip every 20 seconds, what went well between rounds, a review after the walk.
-
8
jobs for Gemini
As Maya, the AI therapist: coach, review, plan, notes, chat, reward and voice.
-
3
rounds a walk
Flowers on a beat, alternating feet, each round a little faster.
-
2
cameras
One above the belt finds the shoes. A phone at the side measures the walk.
-
20 s
between tips
One short tip at a time, and only inside the therapist's limits.
Why
Home exercises that do not get done
Children and teens who are given movement exercises do most of them at home, and home programmes fall apart. Across 13 studies, 34% to 79.2% of parents of children with cerebral palsy did not keep up the home exercise programme (PLOS ONE, 2024). About 1 in 345 children in the United States has cerebral palsy (CDC), and 5% to 6% of school-aged children have developmental coordination disorder (CMAJ, 2015).
Games help when they are part of therapy: across 12 trials, active video games improved gross motor skills in children with developmental disabilities, and the authors ask for caution. What a game at home is missing is someone watching every step. We built an AI therapist to do that, and hid it inside a game.
How a session works
Seven steps, the last one is memory
- Set up The therapist or a parent picks the player, sets safe limits (pace, flowers, rounds, session length) and chooses which AI features may be used.
- Walk and play The player walks on the pad while the game is projected in front of their feet. Gus the zombie has to get home before sunrise; the player steps onto the flowers in time with the beat.
- See the feet A camera above the belt finds each shoe, so the game knows where every foot lands. Perceptron can double-check each step the game scores.
- Measure the walk A phone at the side films the walk. MediaPipe measures knee bend, stride, step time and left–right balance while they play.
- Coach live Gemini puts a short tip on screen every 20 seconds and, in the rest between rounds, says out loud what went well and one thing to fix.
- Review After the walk Gemini writes a review from the numbers: what went well, what to work on, suggested settings, and a note each for the player, the parent and the therapist.
- Remember Backboard keeps a memory for each player, so the next session starts with last time's focus.
Best Use of Gemini API
Maya, eight jobs
Gemini is Maya, the AI therapist, end to end. Seven of her eight jobs run on a small fast model that answers in about a second; the eighth is her voice.
-
A cue during the walk
One short line from the live numbers. Any change to the game is clamped to the therapist's limits.
-
Analysis between rounds
The world stops and Maya says what went well and the one thing to fix. Then 3-2-1, and the next round starts by itself.
-
The session plan
The greeting and today's focus, built from the player's past walks.
-
The end-of-walk note
One note for the player, one for the parent, one for the therapist.
-
The gait review
A structured review of the side camera's numbers. Every finding must cite a measured metric.
-
A chat with photos
The walk's numbers, its review and up to four annotated leg photos go in as images, so Maya answers about what she sees in your own steps.
-
The reward pick
After a completed walk, the reward from the parents' wishlist that fits the effort.
-
The voice
Every spoken line is Gemini text-to-speech, cached per line, with the player's name taken out.
Two rules shaped all eight. The therapist sets the limits and Maya cannot exceed them. And nothing depends on her: without a key every job falls back to fixed rules and the browser's own voice.
The rest of the stack
Each for one thing
- Gemini
- Backboard
- Perceptron
- MediaPipe Pose
- YOLO-World
- Phaser
- FastAPI
- Next.js
- Tailscale Funnel
- Python
- TypeScript
- Backboard
- Memory across sessions: one assistant per player. After every saved walk the result is written to it, and the next plan, greeting and coach context read it back.
- Perceptron
- A second pair of eyes. After each flower, a few timestamped frames of the feet: did the named foot really land on the target? Feet and lower legs only, never a face, and only with consent.
- MediaPipe Pose
- The phone at the side: knee bend, stride, step time and left–right balance, a live skeleton for the dashboard, and annotated photos of notable steps.
- YOLO-World
- The camera above the belt: it finds each shoe, run locally, so the game knows which flower each foot landed on. A bright-blob fallback covers a dark room.
- Phaser
- The projected game: the scenes, the flowers, scoring at the hit line, the coach's lines and the end-of-walk screen.
- FastAPI
- The game server: it relays camera events over a WebSocket, stores players, limits and sessions, and runs every AI call with a rule-based fallback.
- Next.js
- The dashboard for the player, the parent and the therapist: live cameras, game controls, each walk with its review, Maya's chat, settings and consent.
- Tailscale Funnel
- The public link, so judges and phones outside our network could open the dashboard.
The rig
The hard part was physical
Every time we rebuilt the rig, five things had to line up again: the projector's four corners on the belt, the overhead camera's calibration to the same field, the shoe detector's sensitivity to the room's light, the phone at knee height with the whole body in frame, and the browser's permission to play sound.
The moment it paid for itself: the first time a projected flower splashed under a real foot on the walking pad.
-
Walking in place lands every step in its own lane, so plain walking scored everything.
A step counts only for a timed lift above the player's own normal step.
-
One camera cannot measure a millimetre of toe clearance.
Measure what it can, centimetre lifts and step timing, and say so.
-
A projector of about 150 lumens disappears in a lit hall.
High-contrast art on a white board, inside a cardboard booth lined black.
-
Gemini's voice needs about four seconds to arrive, and a safety timer cut lines mid-sentence.
The timer is sized to the line being spoken.
The weekend, in frames
The booth
Three cardboard walls and a roof, lined with black bags. A white board for the picture, a hole for the lens, a lower one for the phone.
What it does not claim
A tool for the therapist, not a replacement
It is not a medical device and it does not replace therapy. The wording in the app stays with coordination, balance, strength and general mobility; there are no disease names and no recovery claims in it. On the walking pad it is for teens and adults within the pad's rated age, with a person beside them, a low speed and a stop button.
The side camera's numbers and the annotated leg photos reach Gemini only with the family's consent, and the player's name is taken out of anything spoken. Perceptron sees pictures of feet and lower legs only. The server keeps no video.
My part
What I did
The rig and the room were mine: the projector in its cardboard booth, the walking pad, the camera above the belt and the phone at the side, and lining all of it up again every time we rebuilt it. So were the game on the floor and the server that joins the parts, the Devpost page and the demo video.
My teammates built the shoe detection and the side-camera gait analysis, the Perceptron checks, Gemini's voice and the review.