Phishing URL Detection using Machine Learning

Phishing remains the cheapest way to breach an organization — one convincing fake login page, one clicked link. Blocklists cannot keep up with the churn of new phishing domains, so content-agnostic URL classification is the practical first filter: decide from the URL string alone, before any page loads. This project engineers 30 such features — URL length, subdomain depth, hyphen and digit patterns, suspicious TLDs, sensitive keywords, and edit-distance to the top-500 brands for typosquat detection — and trains an XGBoost classifier on PhishTank phishing URLs versus Tranco legitimate domains. Evaluation uses a time split (train on older URLs, test on newer) so the score reflects drift, not memorization. The web demo scans any pasted URL, shows the risk gauge, explains every feature's contribution, and keeps a scan history.

1. Phishing URLs are collected from the PhishTank community feed; legitimate URLs are sampled from the Tranco top-1M list with dead domains filtered.
2. The corpus is split by first-seen date: older URLs for training, newer for testing, so evaluation measures drift robustness.
3. 30 features are extracted per URL: lexical (length, entropy, digit ratio, hyphen/dot counts), host (IP literal, subdomain depth, TLD rarity, punycode), path (depth, query count, sensitive keywords), brand (min edit-distance to top-500 brands), reputation (domain-age bucket, redirect chain).
4. An XGBoost classifier (500 trees, max depth 8) is trained with stratified 5-fold cross-validation and early stopping.
5. The time-split test set is scored once: precision, recall, F1, ROC-AUC and the precision-recall curve are exported.
6. Validation-fold analysis picks the block (≥70) and review (40–70) thresholds for the risk policy.
7. In the web demo, a pasted URL goes through the identical 30-feature extractor and the saved model returns the risk score with the full feature breakdown.

Tech stack:

• Python 3, XGBoost, scikit-learn
• PhishTank feed + Tranco list
• pandas, NumPy, Matplotlib
• tldextract (URL parsing)
• Jupyter Notebook (training)
• HTML/CSS/JavaScript (scanner demo)

Dataset & model details

• Dataset: Phishing URLs from the PhishTank community-verified feed; legitimate URLs from the Tranco top-1M domain list (filtered to active domains). Build target approximately 110,000 labeled URLs (50k phishing / 60k legitimate); time-split by first-seen date.
• Task: Binary classification; input = 30 engineered URL features, output = phishing probability (0–100 risk score).
• Model: XGBoost — 500 trees, max depth 8, learning rate 0.05, subsample 0.8, early stopping on validation log-loss.
• Metrics: F1 ~96% (design target for the built-to-order training run), precision, recall, ROC-AUC, PR curve. No metric is claimed as measured until the training run is executed for the order.

• [30-feature engineering] Lexical, host, path, brand-impersonation and reputation features extracted from the URL string alone — no page fetch needed.
• [Live URL scanner demo] Paste any URL for an instant risk score with a per-feature breakdown showing exactly which signals fired.
• [Typosquat detection] Edit-distance of the host against the top-500 brands catches amaz0n-style impersonation.
• [XGBoost classifier] 500 gradient-boosted trees tuned with stratified cross-validation; feature importance exported for the report.
• [Time-split evaluation] Train on older URLs, test on newer ones — measures robustness to phishing drift honestly.
• [Risk policy] Score thresholds map to block / step-up / allow actions, tuned on the validation fold.
• [Scan history] The demo logs every scanned URL with its verdict and top signal.

• Complete training & evaluation Jupyter notebook
• Trained XGBoost model file with the 30-feature extractor
• URL scanner web demo (risk score, breakdown, history)
• Feature-importance chart, PR curve and threshold analysis
• Project report PDF (background, phishing taxonomy, feature engineering, results)
• PPT presentation for final review
• Viva Q&A preparation document (gradient boosting, feature design, drift, thresholds)

• ~96% F1 is a design target for the training run, stated honestly — the report documents the actual achieved figure.
• URL-only signals cannot catch phishing on compromised legitimate domains; content analysis would be a second stage.
• Brand-impersonation detection covers the top-500 brands; obscure-brand phishing scores lower.
• Phishing tactics drift — the time-split evaluation quantifies degradation on newer URLs honestly.
• URL shorteners hide the destination; the demo flags them as suspicious rather than resolving them.