Speech Command Recognition using CNN (Google Speech Commands Dataset)

Voice interfaces live or die on keyword spotting: the tiny model that hears "stop" or "go" and reacts in milliseconds, on a microcontroller-class budget. Getting there is a genuinely hard ML problem — one second of 16 kHz audio is 16,000 samples of speaker variation, room noise and microphone colouration, and the model must be small enough to run on device. This project tackles it the way industry does: audio is converted to log-Mel spectrograms (the time-frequency representation speech models actually learn from) and classified by a compact CNN trained on the Google Speech Commands dataset — 105,829 real one-second utterances of 35 words from 2,618 speakers. Training covers the standard keyword-spotting recipe: noise mixing, time shifts and SpecAugment masking. An audio console demo makes the pipeline visible: record or fire a sample, watch the waveform and spectrogram, and see the 35-way softmax output.

1. The 105,829 one-second, 16 kHz utterances of the Google Speech Commands dataset (v0.02) are loaded with its official train/validation/test lists.
2. Each clip is converted to a 40-band log-Mel spectrogram with a 30 ms window and 10 ms hop, giving a 40 × 98 input image.
3. Training augmentation applies background-noise mixing, random time shifts and SpecAugment time/frequency masking.
4. The compact CNN — three conv blocks → global average pooling → dense(128) → softmax(35) — is trained with Adam and categorical cross-entropy.
5. The held-out test split is evaluated once: top-1 accuracy on all 35 words and on the standard 12-command core subset, plus per-class metrics.
6. The model is exported to TFLite and profiled for size and latency.
7. In the audio console demo, a recorded or sample utterance is turned into the same spectrogram and run through the model, showing waveform, spectrogram and softmax output.

Tech stack:

• Python 3, TensorFlow/Keras
• librosa (audio & Mel features)
• NumPy, Matplotlib, scikit-learn (metrics)
• Jupyter Notebook (training & evaluation)
• TensorFlow Lite (edge export)
• Web Audio API + canvas (console demo)
• Google Speech Commands dataset (v0.02)

Dataset & model details

• Dataset: Google Speech Commands v0.02 (Pete Warden, Google, 2018) — 105,829 one-second 16 kHz mono utterances of 35 command words, recorded by 2,618 volunteers on phones in varied conditions; ships official train/validation/test lists. Standard 12-command core subset (yes, no, up, down, left, right, on, off, stop, go + silence + unknown) used for the subset metric.
• Task: 35-class audio classification; input = 40 × 98 log-Mel spectrogram of a 1-second clip, output = probability distribution over the 35 command words.
• Model: Compact CNN (~200k parameters): Input(40×98×1) → Conv2D(64, 3×3, ReLU)+BN+MaxPool+Dropout(0.2) → Conv2D(64, 3×3, ReLU)+BN+MaxPool+Dropout(0.2) → Conv2D(32, 3×3, ReLU)+BN+GlobalAvgPool → Dense(128, ReLU)+Dropout(0.5) → Dense(35, softmax). TFLite export included.
• Metrics: 35-way top-1 accuracy ≈ 90% and 12-command subset accuracy ≈ 95% (design targets for the built-to-order training run), per-class precision/recall, confusion-pair analysis. No accuracy is claimed as measured until the training run is executed for the order.

• [Compact CNN for keyword spotting] Three convolution blocks (64/64/32 filters) with batch-norm, pooling and dropout, ending in a 128-unit dense layer and 35-way softmax — roughly 200k parameters, designed for on-device budgets.
• [Log-Mel spectrogram front-end] 1-second clips converted to 40 × 98 log-Mel spectrograms (30 ms window, 10 ms hop) — the standard speech representation, computed identically in training and inference.
• [Audio console demo] Record from the microphone or fire sample utterances; the console renders the live waveform, the spectrogram heatmap and the top-5 softmax predictions.
• [Keyword-spotting augmentation] Background-noise mixing from the dataset's own noise clips, ±100 ms time shifts and SpecAugment-style masking, all implemented in the training notebook.
• [TFLite export] The trained model is converted to TensorFlow Lite with a documented size/latency profile for edge deployment.
• [Full training notebook] Data loading with the dataset's official SHA-1 train/validation/test lists, augmentation, training loop and evaluation in one reproducible notebook.
• [Confusion-pair analysis] Per-class metrics with an honest breakdown of where errors concentrate (no/go, three/tree) — the classic keyword-spotting failure mode, and a strong viva topic.

• Complete training & evaluation Jupyter notebook
• Trained CNN (TensorFlow SavedModel + TFLite export) with feature-extraction code
• Audio console web demo (record / sample utterances, waveform + spectrogram + predictions)
• Per-class metrics, confusion-pair analysis and training plots
• Project report PDF (background, speech-feature theory, architecture rationale, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (spectrograms, CNNs for audio, augmentation, on-device ML)

• Isolated one-second words only — continuous speech needs a streaming decoder, listed as future scope.
• English commands; heavily accented or children's speech is harder despite the 2,618-speaker spread.
• Heavy background noise degrades accuracy; the included VAD gate helps but is not perfect.
• Phonetically close pairs (no/go, three/tree, up/off) are the expected confusion points — documented, not hidden.
• 90% / 95% are design targets for the training run, stated honestly — the report documents the actual achieved figures after training.
• The browser console simulates CNN outputs for the demo interface; the real trained weights ship with the project.