Source Code Plagiarism Detector using Token-Based Comparison (Python)

When a programming lab collects sixty assignments, checking originality by reading every file is slow and unreliable, and plain text-comparison tools miss the tricks students actually use: reformatting code, renaming every variable, or stripping comments to make a copy look original. Faculty need a tool that compares what the code says rather than how it is typed. This project is a Python-based detector that tokenizes source files, removes comments, docstrings, blank lines and whitespace, replaces identifiers with positional placeholders, and converts the normalized token stream into k-gram fingerprints using a winnowing-style selection. Pairwise similarity is computed from shared fingerprints on a 0–100 scale, pairs above an adjustable threshold are flagged, and matching blocks are shown side by side with highlighting. A batch mode scans a whole submissions folder and exports a CSV report, so a full class set can be reviewed in one sitting. The design mirrors the token-based approach used by academic originality tools, giving the student a genuine algorithms project with a real user — the faculty reviewer.

1. Submissions are collected from the selected folder as plain Python (.py) text files; the batch scanner walks subdirectories up to 3 levels deep so one folder per student works without renaming anything.
2. The tokenizer strips comments, docstrings, blank lines and whitespace, then replaces every identifier with a positional placeholder (first distinct name VAR1, second VAR2) while Python keywords, operators and literals are preserved exactly.
3. The normalized token stream is split into overlapping k-grams of 5 tokens each, and every k-gram is hashed into a numeric fingerprint with a deterministic string hash.
4. A winnowing pass keeps the minimum hash inside every sliding window of 4 consecutive k-grams, producing a compact fingerprint set per file — approximately 25% of the raw k-gram count for typical short assignment code.
5. Similarity for each pair is computed as shared fingerprints divided by the smaller file's fingerprint count, scaled to 0–100; any pair scoring above the threshold (default 70) is flagged for review.
6. The batch view lists all pairs sorted by score with a color-coded pairwise matrix; clicking a flagged pair opens the side-by-side view with matching token blocks highlighted, and the flagged list exports to CSV with one row per pair.

Tech stack:

• Python 3 (standard library only, no external dependencies)
• Custom Python tokenizer · comment/docstring/whitespace stripper
• Identifier normalization pass (positional placeholders)
• Overlapping k-gram hashing (5 tokens per k-gram)
• Winnowing-style fingerprint selection (window of 4 k-grams)
• Pairwise similarity scorer (0–100 scale)
• Side-by-side diff renderer with block highlighting
• CSV report export · batch folder scanner

• [Comment and whitespace stripping] The tokenizer removes comments, docstrings, blank lines and whitespace before comparison, so two programs with different formatting still produce identical token streams and trivial formatting tricks cannot hide copied code.
• [Identifier normalization] Variable, function and class names are replaced with positional placeholders (the first distinct name becomes VAR1, the second VAR2, and so on) while keywords, operators and literals are kept, so renaming every identifier — the most common obfuscation — still leaves a recognizable pattern.
• [Winnowing-style fingerprinting] The normalized token stream is split into overlapping k-grams of 5 tokens, each hashed to a numeric fingerprint, and a sliding-window selection keeps the minimum hash in every window of 4 k-grams. The compact fingerprint set per file makes pairwise comparison fast even for a whole class of submissions.
• [Pairwise similarity scoring] Every file pair receives a deterministic similarity score from 0 to 100, computed as the fraction of fingerprints shared relative to the smaller file. Identical logic renamed throughout still scores near the top, while independently written solutions score low.
• [Configurable flagging threshold] Pairs scoring above the threshold (default 70, adjustable from 0 to 100) are automatically flagged for faculty review, so the reviewer can tighten or loosen the net without re-running the analysis.
• [Side-by-side diff highlighting] Flagged pairs open in a comparison view that highlights the matching token blocks in both files, letting the reviewer see exactly which lines overlap instead of judging from a bare number.
• [Batch folder processing] Point the tool at a submissions folder and it tokenizes, fingerprints and compares all pairs in one run, then exports a CSV report with pair names, scores and flags — a class set of about 60 short submissions is expected to process in under 60 seconds on a typical laptop (design target).

• Complete Python source code (tokenizer, normalizer, fingerprinter, scorer, batch runner, viewer)
• Command-line interface for single-pair and batch-folder comparison
• Side-by-side comparison viewer with matching-block highlighting
• CSV report export for batch runs
• Sample submission sets for demonstration (original, renamed-copy, reformatted-copy, independent solution)
• Setup and usage documentation (installation, threshold tuning, interpreting scores)
• Project report PDF (background, token-based comparison theory, methodology, worked examples)
• PPT presentation for final review
• Viva Q&A preparation document (tokenization, k-grams, winnowing, threshold trade-offs)