Image Captioning using BLIP

Images are everywhere, yet getting a machine to describe what it sees in plain language remains a hard problem. Captioning sits at the intersection of computer vision and NLP: the model must detect objects, grasp actions and relationships, and express that understanding in a fluent sentence. It powers accessibility tools for visually impaired users and automatic alt-text for the web. Most student captioning builds either copy a weak classical encoder-decoder pipeline or call a closed third-party API, leaving nothing to train, evaluate or explain. A self-contained, trainable vision-language project closes that gap: a transformer pre-trained on image-text pairs, fine-tuned on an open dataset with five human captions per image, and evaluated with metrics computed by the student's own build. That combination — open model, open data, open evaluation — turns a flashy demo into a defensible deep-learning experiment with genuine metric analysis.

Frequently asked questions

1. Why BLIP instead of a classic CNN+RNN captioning pipeline? BLIP is a unified vision-language transformer pre-trained on web-scale image-text pairs, so it starts with a much stronger joint understanding of images and language than an encoder-decoder trained from scratch on COCO alone — and the Hugging Face pipeline makes fine-tuning straightforward.
2. What dataset does the build use? MS COCO Captions: 123,000 images with 5 human-written captions each. The fine-tuning notebook uses the standard train split; evaluation runs on the validation split.
3. How are the captions evaluated? The evaluation notebook generates captions for the validation split and scores them with BLEU-4 (n-gram precision against the 5 reference captions) and CIDEr (a consensus-based metric that weights informative n-grams). The scores in the report are produced by the build's own model.
4. Does the demo need a GPU? No — inference runs in seconds on a regular laptop CPU. Only fine-tuning benefits from a GPU, and the notebook includes cloud-GPU guidance.
5. Is this project suitable for a final-year project? Yes. It suits B.E./B.Tech students in Computer Science, AI/ML and Data Science, demonstrating multimodal transformers, cross-attention, beam search decoding, BLEU/CIDEr evaluation and a live web demo.
6. What will I receive? Complete source code (fine-tuning notebook, evaluation notebook, Streamlit app), fine-tuned BLIP weights for CPU inference, COCO Captions download and preprocessing scripts, project report PDF, PPT presentation, viva Q&A document and a setup guide.

Dataset & model:
Dataset name: MS COCO Captions.
Source: publicly released vision-language benchmark.
Task: image captioning (vision-to-language generation).
Classes: not a classification task — the dataset provides 5 human-written captions per image (123,000 images).
Model: Salesforce BLIP (ViT-B), fine-tuned for image captioning via Hugging Face Transformers.
Input: uploaded image, resized, center-cropped and normalized for the ViT encoder.
Prediction: token-by-token caption generation through a cross-attention decoder.
Output: top caption, alternative captions with sequence scores, attention heatmap.
Evaluation metrics: BLEU-4 and CIDEr — computed by the evaluation notebook on the validation split during the build.
Design target: ~1.0+ CIDEr on the COCO validation split. This is a design target, not a pre-measured claim; the notebook computes the actual score during the build.

Working:

1. Training phase: the pre-trained BLIP (ViT-B) model is fine-tuned for captioning on the MS COCO Captions training split; the image is preprocessed with resize, center-crop and normalization for the ViT encoder.
2. Evaluation phase: the fine-tuned model generates captions for the validation split; the evaluation notebook computes BLEU-4 and CIDEr with the standard COCO evaluation tools.
3. Inference phase: the user uploads an image (or picks a bundled sample) in the Streamlit app.
4. BLIP's vision transformer encodes the image into visual embeddings, which a cross-attention decoder grounds into language, generating the caption token by token with beam search (or nucleus sampling) until the end-of-sequence token.
5. The app displays the top caption plus runner-up alternatives with their sequence scores, and an attention heatmap overlays the image regions that drove the key words.
6. First-time setup downloads the base model weights from Hugging Face, so an internet connection is needed once before the offline demo runs.

Specifications:
Model | Salesforce BLIP (ViT-B), fine-tuned for image captioning
Dataset | MS COCO Captions — 123k images, 5 human-written captions each
Evaluation | BLEU-4 and CIDEr computed by the notebook on the validation split during the build
Design target | ~1.0+ CIDEr (design target, not a pre-measured claim)
Decoding | Beam search (default) with nucleus sampling option
Inference time | Seconds per image on CPU
Framework | PyTorch, Hugging Face Transformers
App | Streamlit upload-and-caption web app
Language | English captions only

[Natural-Language Captioning] (implemented) — Upload any image and get a fluent English caption in seconds on CPU through the Streamlit app.
[Alternative Captions] (implemented) — Runner-up caption suggestions with confidence scores are shown for comparison alongside the top caption.
[BLIP ViT-B Fine-Tuning] (implemented) — Salesforce's vision-language transformer is fine-tuned on MS COCO Captions; the fine-tuning notebook is included.
[BLEU-4 & CIDEr Evaluation] (implemented) — The evaluation notebook computes both metrics on the validation split using the standard COCO evaluation tools.
[Decoding Options] (implemented) — Beam search is the default; nucleus sampling is available for more varied captions, trading fluency for variety.
[Attention Visualization] (implemented) — Heatmaps show which image regions influenced each generated word — strong explanatory material for the viva.
[CPU Demo] (implemented) — Inference runs in seconds on a regular laptop CPU; no GPU needed for the live demo.
[One-Click Regeneration] (implemented) — Captions can be regenerated instantly for a new sample without re-uploading.

Complete source code (fine-tuning notebook, evaluation notebook, Streamlit app)
Fine-tuned BLIP weights ready for CPU inference
MS COCO Captions download and preprocessing scripts
Project report PDF (vision-language background, BLIP architecture, training setup, metric analysis, error analysis)
PPT presentation for the final review
Viva Q&A preparation document (transformers, cross-attention, beam search, BLEU vs CIDEr, COCO dataset)
Setup guide (environment, dependencies, dataset download steps)

Captions go generic on unusual or out-of-distribution images — the model leans on its COCO training vocabulary for scenes it has rarely seen.
COCO-centric vocabulary: everyday objects and scenes are described well; technical, medical or niche-domain terms are outside the model's vocabulary.
No OCR-in-caption: text visible in the image is described as "a sign" rather than transcribed — reading text needs a separate OCR module.
Beam search favors safe, high-probability phrasing, so captions can feel bland compared to a human's; the sampling options trade fluency for variety.
First-time setup downloads the base model weights from Hugging Face, so an internet connection is needed before the offline demo.
English captions only; all reported figures are design targets from the buyer's own evaluation run.