Hackathon · MHacks 2026, University of Michigan · October 2026
A headset for people who cannot hear where a voice came from and cannot see it unless it is straight ahead. Say “Hello Jax”, and it turns the wearer toward you.
The trigger
Someone across the room says your name. Before you can join the conversation, you have to know where they are. The headset listens for a greeting followed by the wearer's name, works out where the voice came from, and tells the wearer which way to turn: by touch first, then on the display.
Who it is for
Usher syndrome is the most common genetic cause of combined deafness and blindness: hearing loss from birth, then retinitis pigmentosa narrows the field of view to a tunnel. Together they make one ordinary moment hard. You cannot hear which way a voice came from, and you cannot see the person unless they are straight in front of you. By the time you have found who is talking, you have missed what they said.
Every aid we found helps one of the two senses and fails the other: a cochlear implant gives speech but not direction, a visual sign language fails as the field narrows, a phone app answers in audio. Human interpreters do the thing we wanted the headset to do, by touch: who is talking, and where.
Three engines
Offline
A rolling four-second window, transcribed every second. The wake rule: a greeting followed by the name. The name alone never fires, because people say names all the time.
Works with no internet: detection, the buzz and the arrow never need the cloud.
Live
Captions as people speak, and a faster second ear for the wake phrase: the same rule and a shared cooldown, so one phrase buzzes once.
Realtime cannot tell speakers apart, so the mic array does: each direction is a person.
By voice
The catch-up and the judges' transcript: batch Scribe separates speakers by voice; Gemini 3.5 Flash-Lite names them from introductions and writes the summary.
Never on the buzz path. If the cloud fails, the wearer still gets turned toward the voice.
One rule shaped the whole build: the “Hello Jax” path runs on the Pi and works offline. Everything that needs the cloud runs behind it and can fail without touching the buzz.
What the judges saw
A phone app is little use to someone with tunnel vision and hearing loss, so the web app is for the table, not the wearer. On the left, the headset camera with a tunnel-vision overlay the judge can tighten: what they see. On the right, a transcript with speakers told apart by voice and a running summary: what Usher gives them.
Measured
Measured on the hardware during the night, with the mic on the printed band after calibration. Not a benchmark: one room, one team, and a hotspot.
Troubles we hit
Whisper heard “Jax” as “jacks” or “jack”.
Match by sound (Metaphone), and accept “jack” only right after a greeting.
The mic dropped off USB in the first full run and the captions died silently.
An audio watchdog: two seconds of silence from the device, and it re-scans and reopens. Motors moved off the 5 V rail.
Two sources renamed the same person back and forth.
Local rules may only name the unnamed; only Gemini may change a name.
Four people in a busy room became sixteen voices.
A reference clip for every known voice in each request, 1.5 s of clear speech before a new one, quiet talkers ignored, a cap with a nearest-direction fallback.
The first buzz pattern was too strong on the temples.
A soft continuous buzz at 10–20% power that gets softer as the wearer turns, instead of counted pulses.
My part
I led the team. The idea was mine, and the research behind it: before the hackathon I read what people with Usher syndrome actually struggle with, what aids exist and where each one fails, and picked this from three candidates on the morning call.
The hardware was mine to bring and to wire: the reSpeaker array and the Pi, and the motor modules from TouchPoint, my earlier glove. The sound-direction and haptics approach carries over from SPOOT and TouchPoint.
Through the night I reviewed the code we had: sixteen findings with patches, an offline test harness of 144 checks that runs anywhere in three seconds, and the runbook we followed to the judging table. Oliver Lai wrote most of the program on the Pi and tested it on the device; Sajjad Akbari built the web page and the camera server; Matthew Palmer modelled and printed the headband and the mic tray.