Paraphrase Detection using Sentence Transformers

Keyword overlap fails at meaning: "How can I improve my English speaking skills?" and "What are the best ways to get better at spoken English?" share almost no words yet are clearly paraphrases. Sentence transformers solve this by embedding whole sentences into a space where paraphrases sit close together. The Quora Question Pairs dataset (404,290 labeled pairs from the duplicate-question challenge) is the large-scale benchmark; MSRP (Dolan and Brockett, Microsoft Research, 5,801 news pairs) is the classic small, carefully-annotated cross-domain check. This project fine-tunes a siamese bert-base encoder on QQP with a contrastive-style objective, tunes the cosine decision threshold on validation (expected optimum near 0.75), and evaluates accuracy/F1 on a held-out test split — with near-boundary scores flagged "uncertain" instead of forced.

1. Both sentences are encoded by the same bert-base (siamese, shared weights).
2. Mean pooling over token embeddings gives one 768-dim vector per sentence.
3. Cosine similarity between the two vectors measures meaning overlap.
4. Fine-tuning uses a contrastive-style objective: duplicates pulled together, non-duplicates pushed apart (QQP).
5. The decision threshold is swept on the validation split to maximize F1 (≈0.75 expected).
6. Final evaluation reports accuracy and F1 on the held-out QQP test split plus MSRP cross-domain accuracy.

Tech stack:

• Python 3.10, PyTorch
• sentence-transformers, Hugging Face transformers
• Quora Question Pairs (404k pairs)
• MSRP (Microsoft Research)
• Cosine similarity + threshold tuning
• FastAPI/Flask (inference API)
• HTML/CSS/JavaScript (demo UI)
• Git

Dataset & Model Details

• Dataset: Quora Question Pairs — 404,290 labeled duplicate/not-duplicate pairs (~37% duplicates), from the Quora duplicate-question challenge; MSRP — Microsoft Research Paraphrase Corpus (Dolan & Brockett), 5,801 news-domain pairs, for cross-domain checks. Split: train / validation / held-out test.
• Model input: two sentences. Output: cosine similarity score + paraphrase verdict.
• Architecture: siamese bert-base (shared weights) → mean pooling → 768-dim vectors → cosine similarity; contrastive-style fine-tuning on QQP; threshold tuned on validation (≈0.75); 0.65-0.80 flagged uncertain.
• Metric: accuracy + F1 on the held-out QQP test split; MSRP accuracy as cross-domain sanity check — measured after training. No figure is claimed before the fine-tuning run.

• [Siamese SBERT encoder] Shared-weight bert-base with mean pooling → 768-dim sentence vectors.
• [Cosine decision rule] Similarity ≥ 0.75 (validation-tuned) → paraphrase; below → not.
• [Uncertain band] Scores 0.65-0.80 flagged "uncertain" rather than forced into a class — a deliberate product decision.
• [Overlap-vs-embedding contrast] Demo shows Jaccard word overlap next to embedding similarity on the same pair.
• [Tricky-pair presets] Word-sense ambiguity ("bank") and clear negatives document honest failure modes.
• [Threshold sweep] Validation F1 swept across thresholds; the chosen operating point is justified with a curve.
• [Inference API] REST endpoint scoring arbitrary sentence pairs for integration.

• Fine-tuning notebook (siamese training, threshold sweep, evaluation)
• Threshold-tuning script with F1 curve
• Trained encoder weights
• Inference API (pair scoring as JSON)
• Test-set evaluation: accuracy/F1 on QQP + MSRP numbers from the actual run
• Web demo (compare UI, score gauge, tricky-pair presets)
• Project report PDF (methodology, threshold analysis, measured results)
• PPT presentation for final review
• Viva Q&A preparation document

• Sentence-level only; paragraph/document similarity needs aggregation (future scope).
• Word-sense traps ("bank" as river vs. institution) still fool it — documented with examples.
• Negation and subtle scope differences ("all" vs. "some") are known weak spots.
• QQP is question-domain; other domains need the MSRP-style cross-check the report includes.
• GPU recommended for fine-tuning; inference runs on CPU.