ESP32-S3 Voice-Controlled Edge Agent with tinyML Wake-Word Detection

Voice assistants students use every day send raw audio to distant servers, which makes them impossible on privacy-sensitive or offline setups. The engineering alternative is edge voice: a small keyword-spotting model that lives on the microcontroller itself and wakes the system only when it hears its name. This project implements exactly that loop on the ESP32-S3, whose vector instructions are designed for on-chip machine learning. An INMP441 digital MEMS microphone captures audio over I2S; a tinyML wake-word model (from the ESP-DL speech-recognition examples, trained on the Google Speech Commands dataset family) runs in firmware and gates a second small command classifier for a fixed set of control words. Recognized commands switch a 2-channel relay board — a lamp and a small fan in the demo build — while the S3 reports wake events, recognized commands and relay states over Wi-Fi to a companion dashboard. Because the model executes entirely on the device, the project demonstrates real edge AI rather than a cloud-API demo, and the wake-word gating makes the power and privacy story genuinely explainable in a viva.

1. The INMP441 MEMS microphone captures audio continuously and delivers 16 kHz mono PCM frames to the ESP32-S3 over the I2S bus.
2. Firmware windows the audio into ~1-second frames and extracts log-Mel filterbank features, the standard front-end used by the ESP-DL keyword-spotting examples.
3. The wake-word model — a compact neural keyword spotter adapted from the ESP-DL speech-recognition examples — scores each frame for the phrase "Hey ESP"; only a score above the configured threshold proceeds.
4. On a wake detection, the device plays the wake confirmation pattern and switches to the command window: the next ~2 seconds of audio are scored by the small command classifier against the fixed vocabulary.
5. The winning command (above the confidence floor) maps to a relay action — "light on" energizes channel 1, "fan off" releases channel 2 — with debounce and a configurable auto-off timer guarding every state change.
6. Each wake event, recognized command and relay state change is timestamped and published over Wi-Fi to the companion dashboard, which renders the live event log, relay states and the hourly command histogram.
7. If no command above the confidence floor is heard in the window, the device silently returns to wake-word listening — nothing is transmitted and no relay moves.

Tech stack:

• ESP32-S3 DevKit (dual-core Xtensa, vector instructions for on-device ML)
• ESP-DL library · keyword-spotting and speech-commands models
• INMP441 I2S digital MEMS microphone
• Arduino IDE / ESP-IDF firmware (C/C++)
• 2-channel relay module · demo lamp and small DC fan
• Wi-Fi event reporting · single-file HTML/JS companion dashboard
• Google Speech Commands dataset family (training-data reference)

Dataset & model details

• Dataset: Google Speech Commands v2 — 105,829 one-second utterances of 35 short words (plus background-noise samples), released by Google's TensorFlow team. The project adapts the ESP-DL keyword-spotting and speech-commands example models built on this dataset, retargeted to the ESP32-S3 with the project's own wake phrase and command vocabulary.
• Task: Wake-word spotting: input = 1-second 16 kHz mono audio frame, output = wake/not-wake score; followed by closed-vocabulary command classification: input = ~2-second utterance, output = one of the configured command labels plus a reject/unknown class.
• Model: Compact keyword-spotting neural network (depthwise-separable CNN class used by the ESP-DL examples) quantized to run in the S3's SRAM via ESP-DL; the command stage is a small classifier head over the same audio front-end. No large-language-model inference runs anywhere in this project — the "agent" is the wake-then-command loop plus relay actuation.
• Metrics: Wake-word false-accept and false-reject rates per the buyer's own tuning session (buyer-run procedure), command confusion across the fixed vocabulary during the student's calibration run, and recognition latency in milliseconds from utterance end to relay action. No accuracy is claimed as measured until the student's own build is tested; the report documents the procedure and the achieved figures from the buyer-run session.

• [On-device tinyML wake-word detection] A keyword-spotting model runs entirely on the ESP32-S3, listening continuously for the wake word "Hey ESP" and waking the command stage only on detection — no audio leaves the device until it decides to listen.
• [Small on-device command classifier] After the wake word, a second tinyML model classifies a fixed vocabulary of control commands (for example "light on", "light off", "fan on", "fan off", "alert") straight into relay actions.
• [I2S MEMS microphone input] An INMP441 digital microphone feeds the S3's I2S peripheral with clean 16 kHz mono audio, the standard front-end for on-device speech models.
• [Relay actuation with debounce and safety interlocks] A 2-channel relay board switches the demo lamp and fan; firmware applies per-command debounce and a configurable inactivity auto-off so a stuck relay state cannot persist.
• [Live companion dashboard] A web dashboard served from the host shows a running log of wake events and recognized commands, live relay states, and a per-hour command histogram.
• [Status feedback on-device] A status LED and a small buzzer pattern confirm wake detection and command execution, so the demo works even without the dashboard open.
• [Wake threshold and vocabulary configuration] The wake-word sensitivity and the command vocabulary are configurable in firmware, with the tuning procedure documented in the build notes.

• Wired ESP32-S3 prototype with INMP441 microphone, 2-channel relay board and status LED/buzzer
• Firmware: I2S audio capture, tinyML wake-word spotting, command classifier, relay control with debounce and auto-off
• Companion dashboard (single HTML/JS file) with live event log, relay states and command histogram
• Wiring diagram, pin map and power-safety notes
• Wake-threshold and vocabulary tuning procedure for the buyer's own voice and room
• Project report PDF (edge-AI background, tinyML pipeline, methodology, buyer-run results)
• PPT presentation for final review
• Viva Q&A preparation document (keyword spotting, quantization, ESP-DL, edge vs cloud trade-offs)