MCQ Distractor Generation using Transformers

A multiple-choice question is only as good as its distractors: if the wrong options are obviously wrong, the question tests nothing, and writing three genuinely plausible wrong answers per question is slow expert labour. Automating it is a real generation challenge — the model must understand the passage deeply enough to produce options that look right to an unprepared student while remaining unambiguously wrong. This project fine-tunes T5 on the RACE corpus (100k exam questions with human-written distractors) to generate three diverse distractors per question. Diverse beam search keeps the options distinct, a filtering stage drops anything that paraphrases the correct answer, and a ranking step orders survivors by plausibility so the best distractor leads.

1. RACE questions are formatted as: passage + question + correct answer (input) → three human distractors (target).
2. T5-base is fine-tuned on these pairs with a standard seq2seq objective.
3. At inference, diverse beam search decodes multiple candidate distractors per question.
4. Candidates are filtered: near-duplicates of each other and paraphrases of the answer are removed.
5. A plausibility scorer ranks the survivors; the top three are returned with scores.
6. The demo renders the question with the correct answer marked and each distractor scored and explained.
7. Test evaluation reports BLEU-4 vs human distractors, answer-leak rate, diversity and human plausibility ratings.

Tech stack:

• Python 3, PyTorch
• Hugging Face Transformers
• T5-base (Raffel et al.)
• RACE dataset (Lai et al.)
• Sentence embeddings (filtering)
• HTML/CSS/JS demo
• NLTK · sacreBLEU

Dataset & model details

• Dataset: RACE — approximately 100,000 multiple-choice questions from Chinese middle- and high-school English examinations, each with a passage, question, correct answer and human-written distractors (Lai et al., EMNLP 2017).
• Task: Distractor generation; input = passage + question + correct answer, output = three plausible-but-incorrect options.
• Model: T5-base fine-tuned as conditional generation; diverse beam search decoding; embedding-similarity filtering; learned plausibility ranker.
• Metrics: BLEU-4 design target 31.6 vs human distractors on the RACE test split; human plausibility design target 3.8/5; answer-leak rate design target below 4%; distinct-2 diversity design target 0.71. No metric is claimed as measured until the fine-tuning run is executed for the order.

• [T5 distractor fine-tuning] T5-base trained on passage + question + answer → three distractors, using RACE's human-written options as targets.
• [Diverse beam search] Decoding produces three options that differ from each other, not three paraphrases of one idea.
• [Answer-leak filtering] Candidates semantically close to the correct answer are dropped automatically.
• [Plausibility ranking] A scorer orders the surviving distractors so the most believable wrong answer ranks first.
• [Live generation demo] Pick a sample passage and question; the demo generates, filters and ranks three distractors with scores.
• [BLEU + human evaluation] BLEU-4 against RACE's human distractors plus teacher plausibility ratings on a 5-point scale.
• [Exported model] Saved weights with a generate-distractors function for reuse.

• Complete fine-tuning and evaluation Jupyter notebook
• Trained T5 distractor model with inference code
• Live distractor-generation web demo with sample passages
• BLEU, diversity and answer-leak evaluation scripts
• Project report PDF (background, generation theory, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (seq2seq, beam search, BLEU, evaluation design)

• BLEU 31.6 is a design target for the fine-tuning run, stated honestly — the report documents the measured figure after training.
• Generation quality depends on passage quality; vague passages yield vague distractors.
• The model generates English distractors only; other languages are out of scope.
• Human plausibility ratings in the demo are simulated; the report documents the real rating protocol for the order's evaluation.
• GPU inference timing is an expected value, not a measured benchmark.