Image Captioning using CNN-LSTM (Flickr8k)

Describing a photo in words requires a machine to do two hard things at once — see and speak. Image captioning is the task that joins computer vision and natural language processing: a CNN reads the image, an LSTM writes the sentence, and an attention mechanism lets each word look back at the image region that inspired it. The Flickr8k dataset (8,092 photos, five human captions each) is the standard teaching benchmark. This project implements the Show, Attend and Tell architecture — ResNet-101 encoder producing a 14×14 region grid, soft attention over the 196 regions at each decoding step, beam search at inference — trains it with teacher forcing, and evaluates with BLEU, METEOR, ROUGE-L and CIDEr against the five references. The web demo generates captions for photos with hoverable per-word attention weights and ranked beam candidates.

1. Flickr8k's 8,092 images and 40,460 captions are loaded; captions are lowercased, tokenized, and a vocabulary of ~8,789 words is built (min frequency 5).
2. Each image is passed through ResNet-101 once; the 14×14×2048 region features are cached to disk so training never re-encodes.
3. The decoder — an embedding layer, attention LSTM (512 units) and output softmax — is trained with teacher forcing and cross-entropy for 30 epochs.
4. At each step the attention layer computes weights over the 196 image regions, conditioning the next word on the attended visual context.
5. At inference, beam search (width 3) explores candidate sequences and the top-scoring caption is returned.
6. The test split is evaluated once: BLEU-4, METEOR, ROUGE-L and CIDEr against the five reference captions per image.
7. In the web demo, a selected photo's cached region features feed the saved decoder; the app shows the caption with attention weights and beam candidates.

Tech stack:

• Python 3, PyTorch
• ResNet-101 encoder (torchvision)
• Flickr8k dataset (Hodosh et al., 2013)
• NLTK / pycocoevalcap (metrics)
• NumPy, Matplotlib
• Jupyter Notebook (training)
• HTML/CSS/JavaScript (caption demo)

Dataset & model details

• Dataset: Flickr8k — 8,092 photographs with 5 human-written captions each (40,460 captions total), standard train/val/test splits. Released by Hodosh et al. (2013).
• Task: Conditional text generation; input = 14×14×2048 region features from the photo, output = English caption (≤20 tokens) from an 8,789-word vocabulary.
• Model: Encoder: ResNet-101 (frozen, features cached). Decoder: embedding(512) → attention LSTM(512) → Dense(vocab, softmax). Soft attention over 196 regions per step; beam search width 3 at inference.
• Metrics: BLEU-4 ~0.62 (design target for the built-to-order training run), METEOR ~0.54, ROUGE-L ~0.68, CIDEr ~0.91. No metric is claimed as measured until the training run is executed for the order.

• [Encoder-decoder with attention] ResNet-101 encoder (14×14×2048 region grid) + soft-attention LSTM decoder — the Show, Attend and Tell design.
• [Live caption demo] Pick a photo; get a generated caption with per-word attention weights (hover to inspect) and ranked beam-search candidates.
• [Beam search decoding] Beam width 3 at inference with length normalization, replacing greedy decoding's myopia.
• [Full training notebook] Vocabulary building, feature precomputation, teacher-forced training loop and metric scoring in one reproducible notebook.
• [Captioning metrics] BLEU-4, METEOR, ROUGE-L and CIDEr computed against all five reference captions per image.
• [Attention inspection] Per-word attention weights visualized so each generated word traces to its image region.
• [Decoding comparison] Greedy vs beam-search outputs compared on sample images in the report.

• Complete training & evaluation Jupyter notebook
• Trained decoder model with cached encoder features pipeline
• Caption generator web demo (attention weights, beam candidates)
• Metric scores, attention visualizations and sample outputs
• Project report PDF (background, attention theory, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (attention, beam search, teacher forcing, metrics)