Essay Scoring using BERT

Teachers spend hours grading essays, and students wait weeks for feedback that arrives too late to act on. Automated essay scoring has been researched since the 1960s, but transformer models made it practical: BERT's contextual representations capture the coherence, vocabulary and argument structure that distinguish strong writing from weak. The ASAP benchmark (Kaggle / Hewlett Foundation) provides 12,978 student essays across 8 prompt sets with human-resolved scores — the standard dataset the field evaluates on with quadratic weighted kappa. This project turns that into a complete student build: bert-base-uncased fine-tuned with per-set regression heads, wrapped in a demo that scores a sample persuasive essay with trait-level feedback and highlighted suggestions. Because the scoring is inspectable per trait, the report and viva can discuss what the model rewards — and where it fails, such as creative writing outside the training distribution.

1. ASAP essays are WordPiece-tokenized (max 512 tokens); scores are normalized per prompt set since each set uses a different scale (e.g. 2–12 for Set 1, 0–60 for Set 8).
2. bert-base-uncased's CLS embedding feeds a dropout + linear regression head per prompt set (one head each, because scales and genres differ).
3. The model is fine-tuned for about 3 epochs with AdamW (lr 2e-5); every run logs MSE and per-set QWK.
4. At inference, the demo app tokenizes the pasted essay, runs it through the fine-tuned model and rescales the prediction to the set's score range.
5. Trait scores come from auxiliary heads trained on trait-annotated subsets; highlighted spans are mapped from attention-adjacent token contributions.
6. The notebook recomputes per-set QWK on the official splits, so the report's numbers are the student's own.

Tech stack:

• Python 3.10, PyTorch, Hugging Face transformers (bert-base-uncased)
• scikit-learn (QWK computation), NumPy, pandas
• Matplotlib (QWK-by-set charts, score distributions)
• Single-file HTML/CSS/JS demo app (scorer, rubric, evaluation views)
• ASAP Automated Essay Scoring dataset (Kaggle / Hewlett Foundation)
• Trained weights exported from the included fine-tuning run

• [Essay scorer] Paste an essay and get a predicted score on the prompt set's scale, with trait-level breakdowns (content, organization, language, conventions).
• [Trait feedback] Highlighted spans in the essay text with concrete suggestions — grammar fixes, organization notes, word-choice upgrades.
• [BERT fine-tuning pipeline] bert-base-uncased with dropout + per-set linear regression heads, fine-tuned with AdamW; all hyperparameters exposed in the notebook.
• [ASAP dataset view] All 8 prompt sets with genre, grade level, score range and essay counts; why each set needs its own regression head.
• [QWK evaluation view] Quadratic weighted kappa per prompt set, with the human–human agreement reference for context.
• [Architecture view] Tokenization (512 WordPiece tokens), CLS embedding, regression head, MSE-on-normalized-score training — the full stack documented.
• [Fairness framing] Explicit statement that automated scores assist teachers and do not replace human judgment — honest scope examiners respect.
• [Full evaluation logging] Per-set QWK on the official splits — never pre-claimed.

• Complete source code (preprocessing, per-set normalization, fine-tuning, inference, demo app)
• Jupyter training and evaluation notebook (buyer-run procedure: fine-tune, evaluate per set, inspect errors)
• Project report PDF (background, ASAP analysis, methodology, per-set evaluation, error analysis)
• PPT presentation for final review
• Viva Q&A preparation document (transformers, regression heads, QWK, ASAP benchmark, fairness)
• Setup guide (environment, dataset download, training, running the demo)

• One regression head per prompt set — cross-set transfer is poor and the report states this openly; a new prompt type needs its own fine-tuning.
• Trained on US grade 7–10 writing; creative, non-standard or non-native styles outside that distribution score less reliably.
• 512-token limit truncates very long essays; the report documents how truncation is handled.
• Trait-level feedback is assistive, not authoritative — final grades belong to human teachers.
• Fine-tuning BERT needs a GPU for reasonable runtimes; the notebook documents expected runtimes and a smaller-sample quick-run mode.