Duplicate File Finder Desktop Utility

Downloads folders accumulate project-report-final(1).pdf and project-report-final(2).pdf; datasets get copied between Documents and project folders; photo backups duplicate entire camera rolls. Name-based duplicate finders miss the real problem: identical content under different names, and different content under the same name. Content hashing solves it properly — files with identical bytes have identical SHA-256 hashes regardless of name, location or timestamps. This project builds a desktop duplicate finder around that principle. Scanning is two-phase for speed: files are first grouped by size (a unique size cannot have duplicates), and only then are the candidates hashed in chunks so multi-gigabyte files never exhaust memory. Groups are presented sorted by recoverable space with per-file control, and cleanup is safety-first throughout.

1. The user selects which folders to scan (Documents, Downloads, Pictures, project drives) and sets filters.
2. The crawler walks the selected roots recursively, honoring the subfolder and size filters.
3. Files are grouped by size first; only files sharing a size with at least one other file are SHA-256-hashed in chunks.
4. Identical hashes form duplicate groups, each labeled with its hash prefix and total wasted space, sorted largest first.
5. Smart select keeps the newest file per group (or the user hand-picks with checkboxes), then deletes to the Recycle Bin.
6. A summary confirms recovered space, the safety check result and the exported CSV deletion log.

Tech stack:

• Python with hashlib (SHA-256) and os/scandir
• Desktop GUI toolkit (Qt or Tkinter)
• System Recycle Bin/Trash integration APIs
• CSV logging module

• [SHA-256 content hashing] Files are compared by cryptographic hash of their bytes, so renamed copies match and same-named different files don't.
• [Two-phase fast scanning] Size pre-grouping means only real duplicate candidates are hashed — tens of thousands of unique files are never hashed at all.
• [Duplicate groups by wasted space] Identical files are grouped with per-group wasted-space totals, sorted so the biggest wins come first.
• [Smart select] One click keeps the newest copy in every group and selects the rest, with manual per-file checkboxes for full control.
• [Recycle Bin deletion] Deletions go through the system Recycle Bin/Trash — every removal is recoverable for 30 days, never permanent.
• [CSV deletion log] Every deleted file is recorded with path, size and hash in an exportable log for the user's records.
• [System-folder protection] Only user-selected folders are scanned; system directories are categorically excluded and a safety check verifies it.
• [Scan filters] Match by content hash, include subfolders, same-name-only mode, and skip files under a size threshold.

• Complete desktop utility source code (crawler, hasher, review UI)
• Hashing-pipeline documentation (two-phase design, chunked reads)
• Sample duplicate file tree for demonstration
• Setup guide (install, usage, safety notes)
• Project report PDF (background, hashing design, methodology)
• PPT presentation for final review
• Viva Q&A preparation document (hashing, file systems, algorithm complexity)

• Hashing is I/O-bound: the first scan of a very large drive takes time proportional to total bytes read — the two-phase design reduces but does not eliminate this.
• Files that differ by even one byte are different files; near-duplicates (resized images, re-encoded videos) are not detected — perceptual hashing is future scope.
• The safety model trusts the user's folder selection; scanning an external drive the user selected is their explicit choice.
• Recycle Bin recovery windows depend on OS policy and available space, not on this app.
• Network drives can be scanned but hashing speed is limited by network throughput.