Cognitive Load Driven Audio-Visual Speech Enhancement with Neuro-Fuzzy Inference and OpenMHA — IEEE Transactions 2025 Base Paper Project

Audio-visual speech enhancement is speech enhancement that listens and watches at the same time: it uses the sound of the voice and the movement of the speaker's lips together to pull speech out of background noise.

Cognitive-load-driven audio-visual speech enhancement is a hearing-aid system that watches how hard the listener is working and picks the right enhancement for the moment: it reads listening effort, audio SNR and lip-movement cues through a neuro-fuzzy controller and switches between no enhancement, audio-only and full audio-visual enhancement on OpenMHA. This project implements that system for a student build, evaluated on public audio-visual speech data.

Traditional hearing aids mainly amplify sound, but they overlook how hard the listener is working to understand speech, and they ignore the visual cues a speaker's lips provide. The paper by Anwar, Ni, Adetomi, Dashtipour, Gogate, Arslan and Hussain, published in IEEE Transactions on Audio, Speech and Language Processing in 2025, introduces a context- and cognition-aware audio-visual speech enhancement framework for hearing aids: a neuro-fuzzy inference system reads listening effort, audio signal-to-noise ratio (SNR) and visual cues, and selects the right enhancement model for the moment, running as a stream on the openMHA platform. Its reported headline figures, labelled as the paper's own: the audio-visual model reaches STOI 0.90 against 0.37 for the audio-only model, with HASPI around 0.9 for both; in complex mixed noise the audio-visual model reaches STOI 0.92, PESQ 1.85 and HASPI 0.93 against 0.18, 1.6 and 0.7 unprocessed. This project implements that system for a student build, evaluated on the public GRID-CHIME3, LRS3 and DEMAND audio-visual speech data, and adds a clearly separate Indian contribution: a classroom hearing-assist mode retuned for Indian noisy environments, with speaker focus and Hindi–English speech.

This is a built-to-order software project: the cognition-aware enhancement pipeline, demo consoles, report, presentation, and viva Q&A are prepared fresh for you. Starting price: Rs 28,000 (final price confirmed after topic, base paper and compute scope are locked).

Related built-to-order IEEE base-paper projects on this site: vMLLM vision-enhanced multimodal LLM with multi-level visual feature aggregation (https://projectech.in/projects/vmllm-boosting-multimodal-llm-enhanced-visual-features/), M4CXR multitask chest X-ray interpretation with multimodal LLMs (https://projectech.in/projects/m4cxr-multitask-chest-xray-interpretation-multimodal-llms/), 3UR-LLM end-to-end multimodal 3D scene understanding (https://projectech.in/projects/3ur-llm-end-to-end-multimodal-3d-scene-understanding/), and self-supervised vision transformers for low-label industrial defect detection (https://projectech.in/projects/self-supervised-vit-low-label-industrial-defect-detection/).

The Problem

A hearing aid that always runs its heaviest enhancement model drains its battery, while one that never enhances leaves the listener struggling in noise. Listening effort rises exactly when speech is hardest to follow, yet conventional aids do not measure it and do not use the speaker's lip movements — even though lip cues are strongest precisely when the audio signal is weakest. The paper's answer is a controller that spends compute only where it helps: full audio-visual enhancement when effort is high and the signal is poor, audio-only enhancement in the middle, and pass-through when listening is easy. The student build reproduces that reasoning chain on public audio-visual speech data, then faces the variant the paper does not cover: the same controller retuned for Indian noisy environments — classrooms with chatter, traffic outside, ceiling fans and canteen crowds — with Hindi and English speech, without presenting demo output as measured results.

Base paper at a glance

Base paper

• Paper: "Cognitive Load Driven Audio-Visual Speech Enhancement Using a Neuro-Fuzzy Inference Model and OpenMHA Platform", Usman Anwar, Xianpo Ni, Adewale Adetomi, Kia Dashtipour, Mandar Gogate, Tughrul Arslan, Amir Hussain, IEEE Transactions on Audio, Speech and Language Processing, 2025, vol. 33, pp. 3267–3280.
• DOI: https://doi.org/10.1109/TASLPRO.2025.3594296 (IEEE Xplore document 11105086; published online 31 Jul 2025).
• Problem the paper tackles: traditional hearing aids amplify sound but ignore listening effort and lip movements, so they waste battery running heavy enhancement when it is not needed and fail to help when the listener is struggling in noise.
• What is implemented from the paper: a Sugeno fuzzy inference system with Gaussian membership functions (low, medium, high) mapping cognitive load, audio SNR and visual quality to three output classes — no enhancement, audio-only enhancement, audio-visual speech enhancement; a lightweight, customisable deep AVSE model with parallel video and audio encoders, concatenation fusion, a separator module and an audio decoder; integration with the openMHA platform for streaming; and evaluation on the public GRID-CHIME3, LRS3 and DEMAND data with STOI, PESQ, HASPI, MOS, latency and power consumption.
• What is deliberately not claimed: no numeric figure is quoted as this build's result — the student build's figures come only from its own evaluation runs; demo console read-outs are illustrations computed from synthesised waveforms, not measured results; and the RF listening-effort sensor in the paper is custom hardware, while the student build estimates listening effort from scenario inputs in the console.
• Paper-reported results (the paper's own findings, not this build's measured results): the audio-visual model reaches STOI 0.90 against 0.37 for the audio-only model, with HASPI around 0.9 for both; in complex mixed noise the audio-visual model reaches STOI 0.92, PESQ 1.85 and HASPI 0.93 against 0.18, 1.6 and 0.7 unprocessed, with improvements statistically significant (p-value below 0.001 for STOI and PESQ in speech noise). That is the paper's own finding. The student build's report records only the figures its own evaluation runs actually produce.

Own features

These are the student's own contribution, clearly distinct from the base paper:

• [Indian classroom hearing-assist mode] A classroom mode that retunes the neuro-fuzzy controller for Indian noisy environments — classroom chatter, traffic, ceiling-fan hum and canteen crowds — with a speaker-focus option that keeps the teacher's voice in front. The paper evaluates on GRID-CHIME3, LRS3 and DEMAND data, not Indian classrooms, and the console demonstrates it on preset scenarios and phrases, not on a published Indian speech dataset.
• [Hindi–English speech support] The same console demonstrates the enhancement on Hindi and English classroom speech, with preset phrases in both languages, so a question or instruction in Hindi is handled the same way as one in English. No Indian speech dataset is claimed.
• [Interactive neuro-fuzzy and pipeline demo consoles] Demos that make the paper's mechanism visible: a mode selector showing Gaussian membership curves and firing rules, a streaming pipeline console with noisy and enhanced waveforms, and a classroom console with a listening-effort gauge and four noise environments, so a reviewer can see the controller decide instead of taking it on trust. Console waveforms and read-outs are synthesised illustrations, labelled as such on the page.

The pipeline runs as a visible chain, and each stage's output is shown in the demos so a student can explain it in a viva. First, the microphone and camera stream noisy speech and lip video into the openMHA plugin chain. The custom RF sensor in the paper estimates the listener's cognitive load from physiological signals; in the student build the console takes listening effort as a scenario input and says so. The Sugeno fuzzy inference controller fuzzifies listening effort, audio SNR and visual quality with Gaussian membership functions, combines them through its rule base, and defuzzifies the result into one of three modes: no enhancement, audio-only, or full audio-visual. The selected model runs: the audio-visual model passes voice and lip video through parallel encoders, fuses them, separates the target speech from the noise, and decodes clean audio for the earpiece. The Indian classroom mode applies the same controller with thresholds retuned for classroom, traffic, fan and canteen noise, with speaker focus and Hindi–English speech. Evaluation runs the same pipeline on the public datasets and records only the figures the student's own runs produce.

1. Stream noisy speech and lip video from the microphone and camera into the openMHA plugin chain.
2. Estimate the listener's cognitive load (RF sensor in the paper; scenario input in the student console, labelled as such).
3. Measure the audio SNR and the visual-cue quality of the lip video.
4. Fuzzify listening effort, SNR and visual quality with Gaussian membership functions (low, medium, high).
5. Combine the fuzzified inputs through the fuzzy rule base to establish rule strengths.
6. Defuzzify the combined result into one of three modes: no enhancement, audio-only, or audio-visual.
7. Run the selected model — the audio-visual model uses parallel encoders, concatenation fusion and a separator to pull speech out of noise.
8. Decode the enhanced audio and deliver it to the hearing-aid earpiece in real time.
9. Apply the Indian classroom mode for classroom, traffic, fan or canteen noise, with speaker focus and Hindi or English speech.
10. Evaluate on the public GRID-CHIME3, LRS3 and DEMAND data with STOI, PESQ and HASPI, and record only the figures the student's own runs produce.

Tech stack:

• Python 3 · PyTorch · OpenMHA
• Neuro-fuzzy inference · Sugeno FIS · Gaussian membership functions
• Audio-visual speech enhancement · lip-reading cues
• Raspberry Pi 4 · edge deployment
• STOI · PESQ · HASPI evaluation
• HTML · CSS · JavaScript demo consoles
• Matplotlib · evaluation plots
• Git · Jupyter Notebook

Dataset & model details

• Dataset (paper and student build): GRID-CHIME3 speech mixed with bus, street and cafe noises for evaluation; competing-speaker noise from LRS3; non-speech noise from DEMAND, the DNS subset, Freesound and the Clarity Enhancement Challenge. The student build evaluates on GRID-CHIME3, LRS3 and DEMAND only. The paper also evaluates MOS, latency and power consumption alongside STOI, PESQ and HASPI. The student build's figures are its own and are not presented as the paper's results.
• Dataset (demo consoles): Synthesised waveforms and preset Hindi and English phrases drawn for the demo consoles (classroom chatter, traffic, ceiling fan, canteen crowd). They are not recordings or benchmark audio, and they are labelled as such wherever they appear. No benchmark audio is played on this page.
• Dataset (Indian-classroom adaptation): No Indian speech dataset is claimed. The classroom mode is demonstrated in the console on preset scenarios and phrases, and it is labelled as such wherever it appears.
• Task: Cognition-aware audio-visual speech enhancement; input = noisy speech plus lip video plus listening effort, output = enhanced speech in the mode the controller selects.
• Model: The paper's system — Sugeno fuzzy inference controller with Gaussian membership functions, lightweight deep AVSE model with parallel audio and video encoders, concatenation fusion, separator module and audio decoder, streamed through openMHA. No architecture detail beyond the paper's description is claimed for it here. The paper reports the AVSE model at 1.57 GFLOPs and deploys it on a Raspberry Pi 4 Model B (quad-core ARM Cortex-A72, 1.8 GHz, 8 GB RAM).
• Metrics: STOI, PESQ and HASPI on the public datasets, plus MOS, latency and power consumption as in the paper; the student build's report records only the figures its own evaluation runs actually produce, and no figure is claimed before those runs.

Cognitive-load-driven speech enhancement delivers seven working capabilities: a Sugeno fuzzy inference controller that selects the enhancement mode from listening effort, SNR and visual cues, a lightweight audio-visual enhancement model, openMHA streaming integration, evaluation on public audio-visual speech data with STOI, PESQ and HASPI, an Indian classroom hearing-assist mode with speaker focus, Hindi–English speech support, and interactive demo consoles a reviewer can operate. The heavy model runs only when the listener is struggling, so battery is saved where enhancement is not needed, and lip cues are used exactly where the audio signal is weakest.

• [Sugeno fuzzy inference controller] A fuzzy inference system with Gaussian membership functions (low, medium, high) maps cognitive load, audio SNR and visual quality to three output classes — no enhancement, audio-only enhancement, audio-visual speech enhancement — so the system spends compute only where it helps.
• [Lightweight audio-visual enhancement model] A customisable deep network (1.57 GFLOPs in the paper) with parallel video and audio encoders, concatenation fusion, a separator module and an audio decoder, built for real-time use on edge hardware rather than a desktop GPU.
• [openMHA streaming integration] The enhancement runs as a stream through the Open Master Hearing Aid platform, which hosts the processing with low latency, instead of running enhancement offline on recorded files.
• [Public dataset evaluation] The system is evaluated on public audio-visual speech data — GRID-CHIME3, LRS3 and DEMAND — with STOI, PESQ and HASPI, and only the figures the student's own runs produce are recorded in the report.
• [Indian classroom hearing-assist mode] A mode retuned for Indian noisy environments — classroom chatter, traffic, ceiling-fan hum and canteen crowds — with a speaker-focus option that keeps the teacher's voice in front, demonstrated in the console on preset scenarios.
• [Hindi–English speech support] Classroom speech demonstrated in Hindi and English with preset phrases in both languages, so the same enhancement is shown working for the speech students actually hear.
• [Interactive demo consoles] A neuro-fuzzy mode selector with live membership curves and firing rules, an openMHA streaming pipeline console with before-and-after waveforms, and a classroom console with a listening-effort gauge, so the mechanism is inspectable.

• Complete cognition-aware speech enhancement pipeline source code (Sugeno fuzzy inference controller with Gaussian membership functions, lightweight AVSE model with parallel encoders and separator, openMHA streaming integration, public dataset evaluation with STOI, PESQ and HASPI)
• Interactive demo consoles (the web demos shown on this page): a neuro-fuzzy mode selector with live membership curves and firing rules, an openMHA streaming pipeline console with noisy and enhanced waveforms, and an Indian classroom hearing-assist console with four noise environments and Hindi–English speech
• Project report PDF (background, base paper, methodology, experiments, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (audio-visual speech enhancement, fuzzy inference, openMHA, evaluation metrics, edge deployment, limitations)
• Starting price: Rs 28,000 (final price confirmed after topic, base paper and compute scope are locked)
• Specific next step: message on WhatsApp for a quotation stating scope (cognitive-load-driven audio-visual speech enhancement with neuro-fuzzy inference and OpenMHA, evaluated on public GRID-CHIME3, LRS3 and DEMAND data, plus an Indian classroom hearing-assist mode with speaker focus and Hindi–English speech), starting price Rs 28,000, and delivery under the 110-hour work plan as a built-to-order project

Work plan & timeline

Total work plan: 110 hours, split across six phases. Phase 1, base-paper study and environment setup: 15 hours — reading the Anwar et al. paper, setting up Python, PyTorch, and the openMHA pipeline, and preparing the public GRID-CHIME3, LRS3 and DEMAND data. Phase 2, neuro-fuzzy controller: 25 hours — the Sugeno fuzzy inference system with Gaussian membership functions and the three-mode rule base. Phase 3, AVSE model and openMHA streaming: 25 hours — the lightweight AVSE model with parallel encoders, concatenation fusion and a separator, integrated with openMHA for streaming. Phase 4, evaluation on public datasets: 15 hours — running STOI, PESQ and HASPI on GRID-CHIME3, LRS3 and DEMAND and recording only the figures the student's own runs produce. Phase 5, Indian classroom hearing-assist mode: 15 hours — four Indian noise environments, speaker focus, and Hindi and English speech. Phase 6, presentation and viva preparation: 15 hours — PPT, demo walkthrough, and the viva Q&A document.

Support

Support covers setup help for the delivered code, a walkthrough of the system and the demo consoles, and fixes for defects in the delivered code. Guidance is also provided for adapting the report and presentation to the student's college format and for answering guide questions about the base paper, the neuro-fuzzy controller, openMHA streaming, evaluation metrics, and the student's own contribution. Support does not cover re-running experiments on the student's personal machine beyond the setup walkthrough, and it does not invent results for experiments that were not run.