Recommendation System using Collaborative Filtering

Recommender systems decide what billions of people watch, buy and read, yet most students meet collaborative filtering only as a library call on a toy 5x5 matrix. The real problem is messier: the MovieLens 100K matrix is 93.7% empty, new users arrive with zero ratings, blockbusters drown out niche tastes, and a low RMSE does not guarantee a good top-10 list. Matrix factorization — the approach that won the Netflix Prize — handles the sparsity by compressing users and movies into shared latent factors, but the leap from textbook SVD to a trained, evaluated, explainable system is exactly what coursework skips. This project makes that leap concrete: real SVD training on MovieLens 100K with a logged optimization curve, ranking-metric evaluation under a leave-one-out protocol, and a demo that opens the black box — latent taste factors, nearest users, and a plain-language reason attached to every recommendation.

1. MovieLens 100K ratings (1-5 stars, every user has 20+ ratings) are loaded and split into train/test sets; global, user and item bias terms are computed.
2. SVD factorizes the sparse matrix into 50-dimensional user factors, item factors and biases, optimized with SGD (learning rate 0.005, regularization 0.02) for 30 epochs with RMSE logged each epoch.
3. The test split yields the final RMSE; a leave-one-out protocol (hide each user's latest rating, predict it back, rank all unseen items) yields Precision@K, Recall@K and nDCG@K.
4. At serving time a predicted rating is computed as user-factor dot item-factor plus the bias terms, for every movie the user has not rated.
5. Unseen movies are ranked by predicted rating; the top-N are rendered as cards with predicted stars.
6. Each card's explanation is generated from the rated movie with the highest latent-space overlap, and the demo's model view shows the user's factor bars and nearest users.

Tech stack:

• Python 3.10, scikit-surprise (SVD matrix factorization)
• pandas, NumPy (rating matrix handling, evaluation protocol)
• scikit-learn (train/test splits, similarity utilities)
• Matplotlib (training RMSE curve, metric charts)
• Flask demo app (recommend, model-inspection and evaluation views)
• MovieLens 100K dataset (943 users, 1,682 movies, 100,000 ratings)

• [SVD matrix factorization] The 943 x 1,682 rating matrix is factorized into 50-dimensional user and item latent factors plus bias terms, trained with SGD (30 epochs, regularization 0.02) using the Surprise library.
• [Predicted ratings with explanations] Every recommendation shows its predicted star rating alongside a "because you liked X" line drawn from the highest-overlap movie in the user's own profile.
• [Latent taste profile viewer] The demo renders the active user's factor vector as labeled bars (action intensity, drama depth, sci-fi affinity and more), making the abstract math visible.
• [Nearest-user panel] The three most similar users by cosine similarity in latent space are shown per profile, exposing the neighborhood intuition behind factorization.
• [Training curve visualization] Per-epoch RMSE is logged during training and plotted in the demo, so the optimization story is part of the deliverable.
• [Ranking-metric evaluation] Beyond RMSE, the notebook computes Precision@5, Recall@10, nDCG@10 and catalog coverage under a leave-one-out protocol.
• [Multiple user profiles] Three built-in profiles (action/sci-fi fan, drama/romance lover, thriller/crime buff) let examiners compare how recommendations change with taste.

• Complete source code (data loading, SVD training, leave-one-out evaluation, Flask demo)
• Jupyter training notebook (buyer-run procedure: train, log the RMSE curve, evaluate)
• Evaluation report with RMSE, ranking metrics and coverage from your build
• Project report PDF (background, matrix factorization theory, methodology, evaluation, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (SVD, latent factors, SGD, RMSE vs ranking metrics, cold start)
• Setup guide (environment, MovieLens download, running training and the demo)

• Cold start is unsolved: new users and new movies have no ratings, so the model cannot place them — the report documents the popularity/content-based fallbacks instead of pretending otherwise.
• MovieLens 100K is 93.7% empty; on sparser real-world data factor quality drops and RMSE rises, which the report states explicitly.
• Popularity bias: the model over-recommends blockbusters, and niche tastes get drowned out without a re-ranking step (listed as future scope, not implemented).
• Offline metrics are computed on held-out ratings; real user satisfaction needs an online A/B test, which is out of scope for a student build.
• Pure CF recommends "more of the same" and cannot explain recommendations beyond neighborhood overlap — serendipity and content-aware explanations need the hybrid follow-up.