Continual-learning vision-language model — IEEE Transactions 2025 Base Paper Project

A continual-learning vision-language model is a system that learns new image classes in successive sessions while retaining the classes it learned earlier, instead of retraining from scratch on all data every time. The core problem is catastrophic forgetting: when a model is trained only on new classes, its accuracy on older classes typically falls sharply, because the new training updates overwrite the representations the old classes depended on. This matters wherever a vision system must grow over time, since full retraining is expensive and old training data may no longer be available. This project addresses that problem by anchoring each class to a textual prior, the text encoder's embedding of the class name and description, so every class keeps a stable semantic reference while image features are updated. The result is a student build that can be demonstrated session by session, with retention and forgetting measured openly rather than claimed.

This is a built-to-order software project: the training code, session viewer, report, presentation and viva kit are prepared fresh for you.

Base paper at a glance

Base paper

• Paper: Visual Class Incremental Learning with Textual Priors
• Journal: IEEE Transactions on Multimedia, 2025
• DOI: https://doi.org/10.1109/TMM.2025.3543109
• What is implemented from the paper: session-wise class-incremental learning in which new classes arrive in separate sessions; a vision-language backbone in which image features are aligned with text embeddings; textual priors that anchor each class prototype to its class name and description; a small replay buffer of stored exemplars from earlier classes; and evaluation that reports average accuracy across all seen classes together with retention on the earliest sessions, so forgetting is visible instead of hidden behind a single final score.
• What is not claimed from the paper: no accuracy figure, result table or dataset result from the paper is reproduced or quoted here, because the delivered build is evaluated on its own training runs and the report records only the figures those runs actually produce.

Own features

The student's own contribution is clearly distinct from the base paper and is built on top of it. First, a domain session planner lets the team define its own session split, for example campus objects, retail items or laboratory equipment, instead of only using the default dataset split. Second, an uncertainty triage view flags low-confidence predictions after each session and routes them for human review, so the demo shows where the model is unsure rather than only where it is right. Third, a college-format session report is generated automatically after every session, with per-session accuracy, retention, forgetting and the classes most confused with each other, ready to paste into a Phase-1 or Phase-2 review document. These three features are the team's own work: the paper supplies the incremental-learning method, and the planner, triage view and session report supply the application and reporting layer around it.

A continual-learning vision-language model works by learning in sessions, protecting old knowledge while it absorbs new classes, and measuring what survived after every step, and this build follows that flow end to end. First the dataset is split into sessions and a pretrained vision-language encoder provides both image and text embeddings, so a class is represented by images and by words at the same time. Then each session trains on its new classes while textual priors hold earlier classes in place and a small replay buffer refreshes them, which is the mechanism that limits forgetting. After each session the model is evaluated on every class seen so far, not only on the newest ones, and the viewer updates its retention and forgetting charts from those measurements. Finally the own features turn the raw run into usable evidence: the planner records the team's own split, the triage view surfaces uncertain predictions, and the session report summarises the session for review.

1. The CIFAR-100 dataset is split into sessions of 10 classes each, with a held-out test split reserved for evaluation after every session.
2. A pretrained CLIP-style vision-language encoder produces image embeddings and text embeddings; the text encoder is kept frozen during incremental sessions.
3. For each new class, a textual prior is built from the class name and a short description, and the class prototype is anchored to that text embedding.
4. The session is trained on its new classes together with replayed exemplars from earlier classes, using a design setting of 20 exemplars per class.
5. After the session, the model is evaluated on all classes seen so far, and average accuracy, earliest-session retention and forgetting are computed.
6. The session viewer updates its charts and per-class bars, the triage view lists low-confidence predictions, and a session report is generated.
7. The next session repeats the cycle, and the final report compares retention with textual priors switched on against the same run with priors switched off, as a controlled ablation the team can explain.

Tech stack:

• Python 3
• PyTorch
• CLIP-style vision-language encoder
• CIFAR-100 dataset
• NumPy · scikit-learn
• Matplotlib · session charts
• Jupyter Notebook
• HTML · CSS · JavaScript demo viewer

Dataset & model details

• Dataset: CIFAR-100 — 60,000 colour images of 32 × 32 pixels across 100 classes, from the official CIFAR-100 release, used here in a session split of 10 classes per session with a held-out test split for evaluation after every session. A selected subset is used for the demo viewer, and the full split is used for the training runs.
• Task: Visual class-incremental classification; input = a 32 × 32 colour image, output = a predicted class among all classes seen so far, with a confidence score used by the uncertainty triage view.
• Model: A pretrained CLIP-style vision-language encoder with a frozen text encoder; each class is represented by a prototype anchored to its textual prior, and incremental sessions update image-side features while replay exemplars refresh earlier classes.
• Metrics: Average accuracy across all seen classes after each session, earliest-session retention, forgetting measured as the drop from a session's first accuracy, per-class accuracy, and most-confused class pairs. No accuracy figure is claimed in advance: the report records the figures the delivered training runs actually achieve, and the demo viewer shows illustrative values labelled as such.

Work plan (120 hours)

A continual-learning vision-language model is useful only if its learning, its memory and its honesty about forgetting can all be inspected, and this project's features are organised around exactly those three checks. The session viewer shows learning as it happens, the textual-prior panel shows why a class has a stable reference, and the evaluation views show retention and forgetting after every session rather than a single end-of-training number. The own features add the parts a student team actually needs for a college build: a planner for their own session split, a triage view for uncertain predictions, and a session report that turns each run into review-ready evidence. Together these features let a guide or examiner follow the whole story, from the first session to the last, and ask pointed questions about any session in between.

• [Session-by-session incremental learning] New classes arrive in separate sessions of 10 classes each, and the model is updated only on the current session plus a small replay buffer, never by retraining from scratch on all past data.
• [Textual priors for every class] Each class is anchored to a text embedding of its name and short description, giving old classes a stable semantic reference that new-session training cannot easily overwrite.
• [Retention and forgetting charts] After each session the viewer plots average accuracy, earliest-session retention and new-session accuracy, so catastrophic forgetting is measured openly session by session.
• [Small replay buffer] A design setting of 20 exemplars per class from earlier sessions is retained for replay, balancing memory use against retention in a way the team can vary and discuss.
• [Domain session planner (own feature)] The team can define its own session split, such as campus objects or retail items, and the viewer and report follow that split instead of only the default dataset order.
• [Uncertainty triage view (own feature)] Low-confidence predictions after each session are listed for human review, showing where the model is unsure rather than only reporting an average score.
• [Automatic session report (own feature)] Each session produces a review-ready summary with per-session accuracy, retention, forgetting and most-confused class pairs for Phase-1 and Phase-2 documentation.
• [Interactive session viewer demo] A single-file web viewer lets the team step through sessions, inspect per-class accuracy and open the textual-prior panel for any class.

• Complete training source code and evaluation notebook for the continual-learning vision-language model
• Interactive session viewer demo (session stepping, retention charts, textual-prior panel, triage view)
• Project report PDF covering the base paper, methodology, session results and limitations
• PPT presentation for project reviews
• Viva Q&A preparation document covering incremental learning, forgetting, textual priors, replay and evaluation
• College-format session reports after each session
• Base-paper summary note with the verified DOI link for guide approval

Support

• Setup guidance for the training environment and the session viewer demo
• Guidance on running the with-priors and without-priors ablation for the review
• One round of corrections to the report and presentation after guide feedback, within the agreed scope
• Explanation support for viva questions on incremental learning, forgetting and textual priors