Speech Emotion Recognition using MFCC and CNN (RAVDESS Dataset)

Speech emotion recognition is the task of identifying a speaker's emotional state — neutral, calm, happy, sad, angry, fearful, disgust or surprised — from voice alone, and this project answers it with a pipeline that converts speech into Mel-frequency cepstral coefficients (MFCCs) and classifies them with a convolutional neural network trained on the RAVDESS dataset. Humans read emotion from pitch, energy, speaking rate and voice quality, but these cues are hard to hand-code as rules, and raw audio waveforms are too high-dimensional and variable for simple classifiers to use directly. That gap is why speech emotion recognition remains a strong academic project: it forces the builder to handle real signal processing, feature design, model training and honest evaluation on a labelled dataset. RAVDESS gives the project a validated, gender-balanced acted-speech corpus, so every claim about the model can be traced to real recordings instead of invented examples.

Emotion in speech is carried by prosody and spectral shape, not by words: the same sentence spoken angrily or sadly has different pitch contours, energy levels and speaking rates. A classifier that works on raw waveforms must learn all of that from scratch, which is data-hungry and brittle. MFCCs solve the representation problem by compressing each short audio frame into a small vector that captures how energy is distributed across perceptually spaced frequency bands — a compact, proven representation for speech tasks. The CNN then learns local patterns across time and cepstral coefficients, such as rising pitch contours or bursts of energy, and maps them to one of 8 emotion classes. Because RAVDESS recordings are acted rather than spontaneous, the project is explicit about its boundary: it recognises performed emotion well, and generalisation to natural, conversational speech is a stated limitation, not a hidden one.

Audio enters the system as a speech waveform, is sliced into short overlapping frames, and each frame is converted to the frequency domain, warped onto the mel scale that matches human hearing, compressed with a logarithm and a discrete cosine transform into MFCCs; those MFCC sequences are stacked into a two-dimensional time–frequency input that the CNN scans for local emotional patterns, and a softmax layer turns the network's final activations into a probability over the 8 emotions, with the highest-probability class reported as the predicted emotion. Training and inference are kept separate: the CNN learns only from the labelled RAVDESS training split, is tuned on a validation split, and is scored once on a held-out test split, so the reported metrics describe genuinely unseen speakers and utterances rather than memorised ones.

1. The 1,440 audio-only speech files of RAVDESS are loaded (48 kHz WAV), resampled to a single working sample rate, and split by actor so that no speaker appears in both training and test sets.
2. Each clip is framed into short overlapping windows (approximately 25 ms frames, approximately 10 ms hop — design values), and silence trimming and amplitude normalisation are applied.
3. For every frame, a mel spectrogram is computed and compressed into 13 MFCCs, plus first-order delta coefficients, giving a compact time–frequency feature matrix per clip.
4. Feature matrices are padded or trimmed to a fixed length (approximately 3 seconds of speech — design value) so every training example has the same shape.
5. The CNN — stacked convolution–pooling blocks over the time–frequency input, followed by dense layers and an 8-way softmax — is trained with categorical cross-entropy and the Adam optimiser.
6. Validation accuracy is monitored each epoch; the best weights are checkpointed and early stopping prevents overfitting to the training actors.
7. The held-out test split is evaluated once: overall accuracy, confusion matrix and per-class precision/recall/F1 are generated for the report.
8. In the web demo, a selected sample passes through the same MFCC pipeline and the saved model, and the app displays the waveform, spectrogram, MFCC vector and the full 8-class probability distribution.

Tech stack:

• Python 3, TensorFlow · Keras
• librosa (audio loading & MFCC extraction)
• NumPy, Matplotlib, scikit-learn (metrics)
• Jupyter Notebook (training & evaluation)
• HTML5 + JavaScript (speech-sample demo)
• RAVDESS dataset (Ryerson University)

Dataset & model details

• Dataset: RAVDESS — Ryerson Audio-Visual Database of Emotional Speech and Song (Livingstone & Russo, Ryerson University; open-access paper in PLOS ONE). The full corpus holds 7,356 recordings from 24 professional actors (12 male, 12 female); this project uses the audio-only speech subset of 1,440 files (60 trials per actor × 24 actors), 48 kHz WAV, two lexically matched statements ("Kids are talking by the door", "Dogs are sitting by the door").
• Classes: 8 speech emotions — neutral, calm, happy, sad, angry, fearful, disgust, surprised. Each non-neutral emotion is recorded at two intensities (normal, strong); neutral has normal intensity only. Each vocalisation is repeated twice.
• Task: 8-class audio classification; input = MFCC feature matrix of a speech clip (13 MFCCs + deltas over time), output = probability distribution over the 8 emotions.
• Model: CNN over the MFCC time–frequency input: stacked Conv2D–MaxPool blocks (design target: 2–3 blocks, 32/64 filters) → Dropout → Dense → Dense(8, softmax). Exact filter counts and depth are fixed during the build and documented in the report.
• Metrics: Test accuracy, per-class precision/recall/F1, confusion matrix. No accuracy is claimed as measured until the training run is executed for the order; the build targets approximately 65–75% speaker-independent test accuracy (design target, not a measured claim), a realistic band for acted-speech CNN baselines on RAVDESS.

The demo at the heart of this project is a working speech emotion pipeline: a sample selector over RAVDESS-style speech samples, a live waveform and mel spectrogram view, a 13-coefficient MFCC display, and a per-emotion probability panel produced by the CNN classifier. Around that demo sit the training artefacts a viva actually tests — a reproducible notebook, a confusion matrix with per-class metrics, and documented design choices for the feature extractor and the network. Together these give the student three things to show: the signal, the features, and the decision, each inspectable on its own rather than hidden inside a single black box. All eight features below are implemented in the base project; live microphone input, extra emotion classes and a different dataset are optional customisations, not part of the base build.

• [Sample selector with real RAVDESS file references] Eight acted-speech samples, one per emotion, each labelled with its RAVDESS filename convention, speaker, statement and 48 kHz audio-only modality, so the demo stays traceable to the real dataset.
• [Waveform and mel spectrogram views] The selected sample renders as a time-domain waveform and a mel-scaled spectrogram, showing how energy moves across frequency bands over the utterance — the visual bridge between raw audio and features.
• [MFCC coefficient display] The first 13 MFCCs of the analysed frame are shown as a coefficient bar chart, making the actual model input visible instead of hidden behind the prediction.
• [Per-emotion probability panel] The CNN output is shown as a probability distribution over all 8 emotions with the top prediction highlighted, so near-misses (for example happy vs surprised) are visible and discussable.
• [Full training notebook] Data loading from the RAVDESS audio-only speech subset, MFCC extraction, model definition, training loop and evaluation in one reproducible Jupyter notebook.
• [Confusion matrix & per-class report] Precision, recall and F1 for each of the 8 emotions, so the viva discussion can address exactly which emotions the model confuses (typically calm/neutral and happy/surprised pairs).
• [Feature and architecture documentation] MFCC parameters (frame length, hop, coefficient count, mel filterbank), CNN layer layout and training hyperparameters recorded with the reasoning behind each choice.
• [Exported trained model] Saved model weights plus the MFCC preprocessing pipeline, so the demo runs the trained network without retraining.

• Complete training & evaluation Jupyter notebook (RAVDESS loading, MFCC extraction, CNN training, evaluation)
• Trained CNN model file with the MFCC preprocessing pipeline
• Speech-sample web demo (waveform, mel spectrogram, MFCC display, per-emotion probabilities)
• Confusion matrix, per-class metrics and training-curve plots
• Project report PDF (background, MFCC theory, architecture rationale, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (MFCCs, mel scale, CNN layers, softmax, speaker-independent splits, acted vs spontaneous speech)