Meeting Minutes Summarizer from Audio Recordings

Meetings generate hours of audio and almost no usable record: someone takes hurried notes, half the decisions are forgotten, and action items evaporate. Manual minutes are slow and biased toward whoever writes them. This project automates the full chain — speech-to-text, dialogue summarization, and structured extraction — using open models and public meeting data. Whisper transcribes the recording, then a BART summarizer fine-tuned on dialogue summaries (SAMSum) produces the minutes, while a rule-and-model extraction stage identifies action items ("Rahul will share the API spec by Friday") and explicit decisions. Training and evaluation use the AMI Meeting Corpus, the standard public corpus of instrumented meetings with reference summaries, so every claim about quality is grounded in a real benchmark. The demo runs the pipeline on a realistic sample meeting with four speakers.

1. Meeting audio is transcribed with Whisper (base/small), producing timestamped segments that are grouped into speaker turns.
2. Speaker turns are formatted as a dialogue transcript and chunked to fit the summarizer's input length, with overlap to preserve context.
3. The fine-tuned BART model generates an abstractive summary per chunk; chunk summaries are merged into the final minutes.
4. Action items are extracted by scanning for commitment patterns (future-tense verbs, "will", "by ", assignee names) and verified with an NER pass for person names and dates.
5. Decision sentences are detected from agreement patterns and collected with their surrounding context into the decision log.
6. The demo renders all four artifacts — transcript, minutes, action items, decisions — from one pipeline run, with ROUGE scores shown for the summarizer.

Tech stack:

• Python 3.11, openai-whisper (ASR)
• Hugging Face Transformers, BART-large (facebook/bart-large)
• SAMSum dialogue summaries (summarizer fine-tuning)
• AMI Meeting Corpus (evaluation)
• spaCy (NER for owners and dates)
• Flask + HTML/CSS/JS (demo)
• ROUGE scoring (evaluation)

Dataset & model details

• Dataset: AMI Meeting Corpus (Carletta et al., IDIAP/Edinburgh) — 100 hours of instrumented meetings with manual transcripts and reference summaries; plus SAMSum (Gliwa et al., 2019) — ~16k messenger-style dialogues with human-written summaries for fine-tuning. Sources: groups.inf.ed.ac.uk/ami/corpus and Hugging Face datasets.
• Task: (1) ASR: audio → transcript; (2) abstractive dialogue summarization: multi-speaker transcript → minutes; (3) extraction: transcript → action items and decisions.
• Model: Whisper (base/small) for transcription; BART-large fine-tuned on SAMSum for summarization; spaCy NER + dependency patterns for action-item/decision extraction.
• Metrics: Word Error Rate (WER) for transcription on AMI; ROUGE-1/2/L for summaries against AMI reference summaries — both reported as design targets from the training run, never pre-claimed.

• [Whisper transcription] OpenAI's Whisper transcribes meeting audio to timestamped text; the demo shows the full speaker-labeled transcript it works from.
• [Abstractive minutes generation] A BART-large model fine-tuned on dialogue-summary data writes concise minutes in natural prose, not stitched-together quotes.
• [Action-item extraction] Sentences containing commitments are detected and paired with owners and deadlines using named-entity recognition plus dependency patterns.
• [Decision log] Explicit decisions ("we agreed to…", "let's go with…") are collected into a separate decision register with the discussion context.
• [Speaker-aware pipeline] Speaker labels flow through every stage, so minutes attribute statements and action items carry an owner.
• [ROUGE-evaluated] Summarizer quality is measured with ROUGE-1/2/L against AMI reference summaries; the report shows the scores honestly.
• [Live web demo] Upload-style demo walks a sample meeting through transcript → minutes → action items → decisions in one screen.

• Full pipeline source (transcription → summarization → extraction)
• Fine-tuned BART summarizer weights
• Training notebook (SAMSum fine-tuning, AMI evaluation, ROUGE scoring)
• Web demo with a realistic preloaded 4-speaker meeting
• Evaluation report (WER + ROUGE tables on AMI)
• Project report PDF (background, ASR + summarization theory, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (Whisper, BART, ROUGE, abstractive vs extractive)