AI-Generated Text Detection using NLP

ChatGPT and other large language models now write essays, reviews, code explanations and social posts that are hard for a reader to tell apart from human writing — a real problem for academic integrity, content platforms and hiring pipelines. Humans are unreliable judges of authorship: studies behind the HC3 corpus (Guo et al., 2023) found people unaided struggle to separate ChatGPT answers from expert human answers to the same questions. Detectors work because LLM writing leaks statistical fingerprints — flatter, more uniform sentence structure, narrower vocabulary spread, lower burstiness — that a fine-tuned transformer can learn. This project fine-tunes RoBERTa on HC3's paired human/ChatGPT answers across question-answering, finance and medical-dialog domains, then stress-tests the detector on TuringBench's 19-generator benchmark to see how it holds up against models it never trained on. The demo turns the statistics into something inspectable: paste text, get an AI-probability with per-sentence highlighting and the tell-tale metrics behind it.

1. HC3 English question-answer pairs are loaded: human expert answers and ChatGPT answers to the same questions, labeled 0 (human) and 1 (AI).
2. Text is tokenized for RoBERTa while a parallel pipeline computes statistical features — sentence-length distribution, type-token ratio, bigram repetition.
3. RoBERTa-base is fine-tuned for binary classification with the statistical feature vector fused before the classification head, using stratified train/validation/test splits.
4. The held-out HC3 test split is evaluated once: accuracy, F1, AUC and the human-text false-positive rate are recorded.
5. Without retraining, the detector scores TuringBench samples from 19 generators to measure cross-generator robustness for the report.
6. In the demo, pasted text is analyzed with the heuristic tell-tale metrics; the shipped build wires in the fine-tuned RoBERTa model for the real verdict.
7. A threshold sweep in the notebook documents the false-positive/recall trade-off for the report.

Tech stack:

• Python 3, PyTorch
• Hugging Face Transformers
• RoBERTa-base (Liu et al.)
• HC3 corpus (Guo et al., 2023)
• TuringBench (Uchendu et al., 2021)
• scikit-learn (metrics)
• HTML/CSS/JS demo
• Git

Dataset & model details

• Dataset: HC3 (Human ChatGPT Comparison Corpus) — English subset: ~58k human responses and ~26k ChatGPT responses to ~24k questions from ELI5, WikiQA, crawled Wikipedia, Medical Dialog and FiQA, released by Guo et al. (2023). Cross-generator stress test on TuringBench — benchmark samples from 19 language models, released by Uchendu et al. (2021).
• Task: Binary human-vs-AI text classification; input = document text, output = AI-generated probability with per-sentence scores and tell-tale metrics.
• Model: RoBERTa-base fine-tuned for sequence classification, with GLTR-style statistical features (token-surprise statistics, sentence-length uniformity) fused before the classification head.
• Metrics: AUC design target 0.93; F1 design target 0.88; human-text false-positive rate design target below 5%. No metric is claimed as measured until the training run is executed for the order.

• [RoBERTa fine-tuning] RoBERTa-base fine-tuned for binary human-vs-AI classification on HC3's English subset, with the training recipe documented end to end.
• [GLTR-style statistical features] Token-level surprise statistics in the spirit of GLTR (Gehrmann et al., 2019) fused with the transformer signal to expose generation fingerprints.
• [Cross-generator evaluation] The detector is stress-tested on TuringBench texts from 19 different generators, measuring how detection degrades on unseen models.
• [Live paste-and-analyze demo] Paste any text; the demo returns an AI-probability gauge, per-sentence highlighting and the underlying metrics.
• [Per-sentence highlighting] Each sentence is colored by its AI-likelihood so the exact passages driving the verdict can be inspected.
• [Statistical tell-tale metrics] Burstiness, vocabulary richness, repetition rate and sentence-length uniformity shown alongside the verdict.
• [Threshold analysis] The notebook sweeps the decision threshold to document the human-text false-positive vs AI-text recall trade-off.

• Complete fine-tuning and evaluation Jupyter notebook
• Trained RoBERTa AI-text detector with inference code
• Live text-analysis web demo with sample texts
• Cross-generator (TuringBench) evaluation scripts
• Project report PDF (background, detection research, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (RoBERTa, fine-tuning, perplexity, burstiness, false positives)