In this guide
Every voice-assistant or transcription project hits the same wall: turning audio into text reliably. Before 2022, that meant training acoustic models on thousands of hours of labeled speech — out of reach for a student project. OpenAI's Whisper changed the equation: a single open-weight model, trained on 680,000 hours of audio, that transcribes dozens of languages out of the box with no training required.
This guide covers how Whisper works at a practical level, how to run it (locally and via API), which model size to pick, how to handle long audio, timestamps, and multiple languages — and where it falls short.
What Whisper is and why it works so well
Whisper is an encoder-decoder transformer trained on a massive, weakly-supervised dataset: audio paired with transcripts scraped from the internet. The scale and diversity of that data — accents, background noise, music, dozens of languages — is what makes it robust. It was not trained on clean studio audio, so it handles real-world messiness better than models trained on curated corpora.
The architecture at a glance:
| Component | What it does |
|---|---|
| Audio encoder | Converts 30-second log-Mel spectrogram chunks into representations |
| Text decoder | Generates the transcript token by token, conditioned on the audio |
| Special tokens | Control the task: transcribe vs translate, language ID, timestamps |
Two headline capabilities beyond plain transcription:
- Translation: transcribe non-English audio directly into English text (task token switches the mode).
- Language identification: detect which of 99 languages is spoken, no configuration needed.
Note: Whisper is a transcription model, not a real-time streaming model. It processes 30-second windows; true low-latency streaming needs extra engineering (covered below).
Choosing a model size
Whisper comes in five sizes. Bigger is more accurate and slower — the tradeoff is stark:
| Model | Parameters | Relative speed | Good for |
|---|---|---|---|
| tiny | 39M | ~10x | Keyword spotting, drafts on weak hardware |
| base | 74M | ~7x | Quick prototypes |
| small | 244M | ~4x | Good balance for many projects |
| medium | 769M | ~2x | Serious transcription work |
| large-v3 | 1.55B | 1x | Highest quality; needs a decent GPU |
English-only variants (*.en) exist for tiny through medium and are slightly better on English. Rule of thumb: prototype with small, ship with medium if your hardware allows, and only reach for large when accuracy is the project's core claim. There is also distil-large-v3 (a distilled version) and turbo — faster variants worth knowing about when latency matters.
Running Whisper locally
The fastest path uses the faster-whisper library (CTranslate2 backend — much faster than the reference implementation):
from faster_whisper import FasterWhisper
model = FasterWhisper("small")
segments, info = model.transcribe("lecture.wav", language="en")
for seg in segments:
print(f"[{seg.start:.1f}s -> {seg.end:.1f}s] {seg.text}")
Or the Hugging Face transformers pipeline:
from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
result = asr("lecture.wav", return_timestamps=True)
print(result["text"])
Hardware notes: tiny/base run on CPU in reasonable time; small is usable on CPU for short clips; medium and above want a GPU. For a free-GPU workflow, see the Google Colab GPU guide.
Handling long audio
Whisper processes 30-second windows. For a one-hour lecture, you need chunking:
- Naive chunking (split every 30 s) cuts words at boundaries — avoid it.
- Overlapping windows with crossfade: transcribe overlapping 30 s chunks and merge, deduplicating the overlap region.
- VAD first (voice activity detection): split on silence using a VAD model (Silero VAD is the common choice), then transcribe speech segments. This avoids wasting compute on silence and gives natural boundaries.
Both faster-whisper and the transformers pipeline handle long-form transcription with VAD-based chunking built in — use their long-audio modes rather than writing your own splitter.
Timestamps, diarization, and subtitles
Word- and segment-level timestamps come free with most implementations — enough to generate SRT/VTT subtitles:
segments, _ = model.transcribe("talk.wav", word_timestamps=True)
Speaker diarization ("who spoke when") is not built into Whisper. The standard stack pairs Whisper with a diarization model (pyannote.audio is the reference): diarize to get speaker-labeled time ranges, transcribe with Whisper to get word timestamps, align the two. For a meeting-transcription project, this combination is the whole architecture.
Multilingual behavior and its quirks
Whisper auto-detects language, but explicit language="en" (or your target language) is more reliable and faster — detection costs a forward pass. Known quirks:
- Code-switching (mixing languages mid-sentence, common in India) degrades quality; results vary by language pair.
- Low-resource languages transcribe worse than English — check quality on your actual target language before committing.
- Translation mode (
task="translate") produces English text from other languages, but quality trails dedicated translation pipelines for difficult pairs.
Always evaluate on audio representative of your deployment: your microphone, your accents, your background noise. Clean test clips lie.
Whisper vs alternatives
| Option | Strengths | Weaknesses |
|---|---|---|
| Whisper (local) | Free, private, multilingual, robust | Compute cost, not streaming |
| Cloud APIs (various STT services) | No infra, often streaming-capable | Per-minute cost, data leaves your machine |
| Wav2Vec2 (fine-tuned) | Excellent when fine-tuned on your domain | Needs labeled data and training |
If your project involves a specific accent or vocabulary (medical terms, regional language), fine-tuning Wav2Vec2 on your data can beat Whisper. For general transcription with zero training, Whisper is the default.
Common mistakes
- Feeding it music or overlapping speech and expecting miracles. Whisper handles noise, but two people talking over each other is genuinely hard — results degrade.
- Wrong sample handling. Whisper expects 16 kHz mono; libraries usually resample for you, but verifying your pipeline's audio preprocessing saves mysterious quality drops.
- Hallucinated repetitions. On silence or noise, Whisper sometimes loops phrases ("thanks for watching" on YouTube rips is the famous example). VAD pre-filtering and the
no_speech_thresholdparameter mitigate this. - Using large-v3 on CPU for a demo. Know your latency budget: time a 1-minute clip on your actual hardware before promising real-time.
- Ignoring the API option for prototypes. If transcription is a feature rather than the research contribution, a hosted API gets you to a working demo in an afternoon.
- No evaluation. Transcribe 10 representative clips, compute word error rate against manual transcripts, and report it. "It sounded fine" is not a metric.
Where to go from here
For domain-specific speech recognition that beats generic Whisper, read speech recognition with Wav2Vec2. If your pipeline needs an LLM to act on transcripts, the prompt engineering guide covers reliable structured outputs from transcripts. For free GPU compute to run larger models, see the Google Colab guide. More audio and speech AI in the AI & Machine Learning branch hub.