How Always-On Voice AI Works: Wake-Word Detection, Voice Activity Detection and On-Device Speech Recognition

How Always-On Voice AI Works: Wake-Word Detection, Voice Activity Detection and On-Device Speech Recognition

On 15 October 2026, Amazon launched Bee — an always-on AI wearable that listens to your day and hands you back summaries, to-dos, reminders and insights. Two devices: the Bee Frontier ($74.99, worn on the wrist or clipped to clothing, about a week of battery) and the Bee Eclipse ($99.99, a puck that sticks magnetically to the back of a MagSafe or Qi2 phone, about three weeks of battery). Both listen when switched on, convert conversations into structured notes, and lean on a physical mic button plus a visible LED as their privacy story — with a $11.99/month Bee Pro subscription unlocking the full feature set.

The product is interesting, but the engineering question underneath it is more interesting: how does a device listen all day without its battery dying by lunch, and without uploading your life to a server? This guide explains the pipeline piece by piece — wake-word detection, voice-activity gating, the digital audio path, offline transcription, and the phone-as-compute architecture — with the honest trade-offs at each stage. We also built the student-scale version of this exact idea: the Bee-Inspired Always-On Voice Companion, an ESP32-S3 puck with an INMP441 I2S mic and a companion phone app where every stage runs locally.

The three jobs of always-on audio

Every always-on voice device does three jobs, and the art is in how rarely it does the expensive ones:

1. Capture — the microphone and audio path run continuously. This stage must sip power, because it never sleeps.
2. Decide — is anyone speaking? Was that the wake word? This runs continuously too, so it must be tiny and cheap to compute.
3. Understand — transcribe the speech, summarize it, extract reminders. This is the expensive step: it wakes up only when the first two stages say something worth understanding happened.

The design rule for the whole field: keep stage 3 asleep as much as possible. Battery life, privacy, and bandwidth all follow from that one rule.

How wake-word detection actually works

Wake-word detection is streaming keyword spotting, and it is a solved-enough problem that it runs on microcontrollers:

• Framing: the audio stream is sliced into overlapping windows (typically around 1 second, advancing every ~100 ms), so the model re-scores the recent past continuously.
• Features: each window is converted to features a small model can read — MFCCs or mel-spectrograms, the same front-end speech recognition has used for decades. Raw waveforms are too variable; features compress the signal to what distinguishes words.
• The model: a tiny neural network, usually a depthwise-separable CNN, outputs the probability that the window contains the wake phrase. After int8 quantization these models fit in well under 100 KB — small enough for MCU flash and SRAM.
• Threshold and debounce: a trigger fires only when the probability crosses a threshold across consecutive windows. One utterance, one trigger — not a burst of five.
• The two error types you must measure: false accepts (the TV says something similar and the device wakes — costs battery and trust) and false rejects (you say the phrase and nothing happens — costs patience). The threshold trades one against the other; no setting eliminates both, and any report that claims zero of both is not measuring honestly.
• Training data: the Google Speech Commands dataset for the base vocabulary, plus your own recorded samples of your phrase in your actual room. Room acoustics matter more than most students expect — a model trained in a quiet bedroom fails in a tiled lab.
• Compute layout: on a dual-core MCU like the ESP32-S3, the wake-word model can own one core while the other handles audio capture and voice-activity gating. That partitioning is the standard student architecture, and it is exactly what our companion build does.

For the full TinyML workflow behind models like this — datasets, quantization, deployment — see the 10 Best TinyML and Edge AI Projects guide, whose keyword-spotting section covers the training pipeline in detail.

Voice-activity detection: the gate before the radio

Voice-activity detection (VAD) answers a simpler question than the wake-word model: is this speech or not-speech? Classic VAD uses signal energy and zero-crossing rates; neural VAD uses a tiny classifier. Either is far cheaper than streaming audio anywhere.

Why VAD is the unsung hero: the radio is the hungriest part of the device. BLE transmit draws on the order of ~100 mA (datasheet-typical for the silicon); the microphone and the inference cores draw a fraction of that. VAD means the radio only wakes for genuine speech windows — silence, fan noise and room tone never cross the air, and the battery lasts meaningfully longer.

Practical note for your build: tune the VAD in the room you will demo in. Air conditioners, ceiling fans and projectors are the classic false-speech sources, and a VAD tuned in a silent bedroom will stream fan noise all day in a lab.

The audio path: from air to bits

The microphone choice shapes everything downstream. The modern student default is an I2S digital MEMS mic such as the INMP441:

• The MEMS element is a microscopic silicon diaphragm whose deflection changes capacitance; the change is digitized on the chip itself, and the result leaves as a 24-bit I2S digital stream (61 dBA SNR, 60 Hz–15 kHz, −26 dBFS sensitivity — manufacturer figures).
• Digital output means no analog stage, no codec chip, and no preamp noise path to debug — the mic connects straight to the MCU's I2S peripheral.
• 16 kHz sampling is the speech standard. Human speech carries its information roughly between 300 Hz and 3.4 kHz; 16 kHz sampling captures that with headroom. Higher rates cost power, memory and BLE bandwidth for no keyword-spotting benefit.
• Mechanical note: the mic grille must face the room, and the enclosure needs a real acoustic port. A sealed 3D-printed case kills high frequencies first and makes every model downstream perform worse — the most common silent failure in student audio builds.

On-device vs cloud transcription

Once a speech snippet exists, someone must turn it into text. Two philosophies:

• Cloud transcription uses large server models — the best accuracy available, but the audio leaves the device. That is the commercial trade: better transcripts in exchange for uploading your conversations and, usually, a subscription.
• On-device transcription (for example the Vosk small English model, roughly 40 MB, running on the phone) keeps every audio frame local. The honest cost is accuracy: expect word errors in noisy rooms, with strong accents, or with overlapping speakers. Privacy and offline operation are the features; transcription quality is the stated cost, and a good report says so openly.

Why the phone does the transcribing, not the puck: model size. A Vosk-class model needs tens of megabytes of RAM; an ESP32-S3 has 512 KB of SRAM. The architecture follows the hardware — capture and detection on the microwatt budget of the puck, understanding on the phone that is already in your pocket.

The phone-as-compute pattern (the Bee Eclipse idea)

Bee's Eclipse is the clearest commercial expression of this pattern: the puck sticks to the phone's back and leans on the phone for compute. Our student build uses the same split — the puck captures and detects, the phone transcribes, summarizes and stores.

The constraint that shapes the pattern is BLE throughput. Short speech snippets cross a BLE link comfortably; continuous stereo audio does not. So the design is: wake word fires, VAD-gated snippets stream for a few seconds each, the phone does the rest, and the phone stays within a few metres. If your requirements ever say "stream everything continuously," you have outgrown BLE and need a different radio — that is a design decision, not a bug.

For the BLE concepts involved — advertising vs connections, GATT, and practical throughput — the BLE basics guide is the right companion read.

Privacy by architecture, not by promise

An always-on listener earns trust through hardware, not terms of service:

• Hardware mute: a slide switch wired in series with the microphone's power supply. Firmware cannot override a power cut — when the switch is open, the mic is unpowered, full stop.
• Mic-rail LED: an indicator wired to the microphone power rail rather than a GPIO pin. Lit means the mic has power; dark means it physically cannot be listening. You can verify the claim with a multimeter, which is precisely the point.
• Local storage: transcripts and summaries in an on-device database, no accounts, no sync. The privacy architecture is the absence of a data path, not a promise about one.
• Consent: record only conversations you are part of, with everyone present aware of it. Laws on recording vary, and "the device was listening" is not a defence. Say the consent rule out loud in your viva — examiners respect it.

Practical: what to measure in your build

A voice-AI build is graded on measurements, not adjectives. Four numbers make the report:

1. Current in each state — multimeter in series with the battery: idle listening, wake-word hit, BLE streaming. These three numbers are your battery-life arithmetic; the low-power battery design guide shows how to turn them into honest day-counts.
2. False accepts per day — leave the puck in a normal room with the TV on for 24 hours and count spurious wake-ups. This is the metric commercial teams actually track.
3. Word error rate — record a fixed set of test sentences, transcribe them, and compare against the script. That percentage is your honest accuracy number; report it instead of adjectives.
4. BLE range walk test — walk away from the phone until snippets start dropping. Write down the distance; it is your demo boundary.

Our Bee-inspired voice companion project implements this exact pipeline — puck, wake word, VAD, BLE snippets, offline transcription and summarization — and its page documents each stage with the same honesty this guide asks for.

FAQ

Does an always-on device record everything, all the time?
No — and the gating is the engineering. Voice-activity detection discards silence and non-speech; the wake-word model then decides whether the remaining audio contains the trigger phrase. In a well-built device nothing streams anywhere until the wake word lands. "Always-on" describes the listening, not the recording.

Why can't the puck do the transcription itself?
Arithmetic: a usable offline transcription model needs tens of megabytes of RAM, and the ESP32-S3 class of MCU has 512 KB of SRAM. The wake-word model fits because it is under 100 KB after quantization; transcription does not, by two orders of magnitude. So the puck detects and the phone understands.

How accurate is offline transcription, honestly?
Below cloud services, by design. In a quiet room with clear speech it is perfectly usable for summaries and reminders; in noise, with strong accents, or with people talking over each other, expect word errors. For a student build the right move is to measure the word error rate on a fixed test set and print it in the report.

Why not just use the phone's own microphone?
For a pure software prototype, you can — and should, for the first version. The puck earns its place with a dedicated always-on audio path that does not drain the phone, plus hardware privacy controls (mute switch, mic-rail LED) a phone cannot offer. Build the phone-mic version first; add the puck when the software works.

Is it legal to record conversations with a device like this?
Only with consent. Build it for your own conversations — study groups, lab sessions, interviews you conduct — with everyone present aware the device is listening. Secret recording can violate privacy law depending on where you are. The hardware mute switch and LED exist so listening is always visible and stoppable.

Limitations

• Keyword spotting covers a handful of phrases in reasonably quiet conditions; similar-sounding words cause false accepts, and no threshold setting eliminates both error types.
• Offline transcription trades accuracy for privacy — the trade is stated openly, not hidden.
• Realistic battery life for always-on listening is measured in days, not weeks: VAD gating trims the radio, but the mic and inference cores never sleep.
• Speaker diarization (who said what) is not in the base build; summaries attribute speech to the conversation, not to individuals.
• BLE range is a few metres in practice; this is a near-phone device, not a room-scale one.
• Nothing here is a certified product: treat the build as a lab prototype with the consent and battery-safety discipline that implies.