Hindi Speech Recognition using Wav2Vec2

Hindi is spoken by hundreds of millions, yet high-quality open Hindi ASR still lags English — a gap final-year students can meaningfully work on. This project takes facebook/wav2vec2-large-xlsr-53, a transformer pre-trained on 53 languages of unlabeled speech, and fine-tunes it on the validated Hindi subset of Mozilla Common Voice with a CTC objective. Students learn how self-supervised speech models work, why SpecAugment matters, how CTC decoding turns frame predictions into Devanagari text, and how a KenLM language model rescores hypotheses. Evaluation uses word and character error rates on a held-out split. The web demo provides the recorder-and-transcription interface with representative outputs.

1. The Common Voice Hindi validated subset is downloaded and resampled to 16 kHz mono.
2. Transcripts are normalized (Devanagari script, punctuation handling) and split 90/10 train/validation.
3. SpecAugment applies time and frequency masking during training for robustness.
4. wav2vec2-large-xlsr-53 is fine-tuned with CTC loss using Adam and learning-rate warmup.
5. A 5-gram KenLM language model is trained on Hindi text for decoder rescoring.
6. The held-out split is decoded with CTC beam search + KenLM; WER and CER are computed for the report.
7. The web demo records audio and displays a representative transcription with word confidences.

Tech stack:

• Python 3, PyTorch
• Hugging Face Transformers
• facebook/wav2vec2-large-xlsr-53
• Mozilla Common Voice Hindi
• KenLM (5-gram LM)
• LibriSpeech-style eval (jiwer)
• HTML/CSS/JS voice demo

Dataset & model details

• Dataset: Mozilla Common Voice (Hindi, hi) — community-contributed read speech, ~37 hours validated, 16 kHz mono with Devanagari transcripts, CC-0 licensed; 90/10 train/validation split for fine-tuning.
• Task: Automatic speech recognition; input = 16 kHz Hindi speech waveform, output = Devanagari transcript.
• Model: facebook/wav2vec2-large-xlsr-53 (315M params: CNN feature encoder + 24-layer transformer) fine-tuned with a CTC character head; KenLM 5-gram rescoring at decode.
• Metrics: Word error rate 18–22% (design target), character error rate 6–9% (design target), real-time factor ~0.3× on CPU (expected). No WER is claimed as measured until the fine-tuning run is executed for the order.

• Fine-tuned Hindi ASR wav2vec2-large-xlsr-53 fine-tuned on Common Voice Hindi with CTC loss — real transfer learning for speech.
• Real speech dataset Mozilla Common Voice Hindi: ~37 hours of validated 16 kHz read speech with Devanagari transcripts (CC-0).
• Full training notebook Data prep, SpecAugment, CTC fine-tuning, checkpointing and WER/CER evaluation in one reproducible notebook.
• KenLM rescoring A 5-gram Hindi language model rescores CTC beam-search hypotheses — the classic ASR second pass, explained in the report.
• Interactive voice demo Single-file web app with recorder, live waveform visualization and word-level transcription display.
• Honest error analysis WER/CER on the held-out split plus error-type breakdown (substitutions, deletions, insertions) for the viva.
• Viva kit Report PDF, PPT and Q&A covering transformers for speech, CTC, SpecAugment and language modelling.

• Common Voice Hindi preparation scripts
• Complete CTC fine-tuning & evaluation notebook
• Fine-tuned wav2vec2 model + KenLM language model
• Interactive voice demo web app with sample utterances
• WER/CER evaluation tables and error-type analysis
• Decoding pipeline documentation
• Project report PDF, PPT presentation
• Viva Q&A preparation document

• Read speech only — conversational, accented or code-mixed (Hinglish) speech degrades WER.
• WER design targets assume clean recordings; noisy audio needs additional augmentation.
• 18–22% WER is a design target, stated honestly — the report documents the actual achieved figure.
• The 315M-parameter model needs a GPU for training; inference runs on CPU in real time.
• The web demo simulates the pipeline with representative outputs; it does not run the 315M model in-browser.