Hand Gesture Recognition using CNN

Touchless control is the natural interface for media players, presentations and smart displays — but static image classifiers miss the motion that defines a gesture. A thumbs-up is not a pose; it is a movement through time. 3D convolutions extend CNNs along the temporal axis, learning spatio-temporal patterns from short video clips instead of single frames. The 20BN-Jester dataset gives this problem a real benchmark: 148,092 crowd-recorded webcam clips across 27 gesture classes, noisy backgrounds included. This project turns that into a complete student build: a 3D-CNN trained on an 8-class Jester subset (matched to a student GPU budget), wrapped in a recognizer demo that classifies a clip, smooths the prediction and fires the mapped command on screen. The report and viva can discuss the genuinely interesting engineering — why temporal smoothing and a confidence gate matter more than raw accuracy for a usable live demo.

1. Jester v1 clips for the 8 selected classes are decoded to 16 uniformly sampled frames at 112×112 grayscale, respecting the official train/valid/test annotation splits.
2. A 3D-CNN (three conv3d+maxpool blocks, global average pooling, dense softmax head) trains with Adam and categorical cross-entropy for about 40 epochs.
3. Training augmentation applies temporal jitter and spatial crops so the model tolerates timing and framing variation.
4. At inference, live webcam frames are buffered into 16-frame clips; predictions are smoothed over a 5-clip window and gated at 0.6 confidence.
5. The recognized gesture is displayed with its confidence score and the mapped command is fired and shown on screen.
6. The notebook evaluates clip-level accuracy, per-class F1 and the confusion matrix on the validation split.

Tech stack:

• Python 3.10, TensorFlow/Keras (Conv3D layers)
• OpenCV (video decode, frame sampling, live capture)
• NumPy, scikit-learn (metrics, confusion matrix)
• Matplotlib (training curves, per-class F1 bars)
• 20BN-Jester v1 dataset (148,092 clips, 27 classes; 8-class subset used)
• Single-file HTML/CSS/JS demo app (recognizer UI, dataset explorer)

• [8-gesture vocabulary] Swiping left/right, thumb up/down, stop sign, two-finger zoom in/out and no-gesture — the Jester classes most useful for media control.
• [3D-CNN classifier] Stacked 3D convolution + pooling blocks over 16-frame 112×112 grayscale clips, trained with augmentation (temporal jitter, spatial crops).
• [Confidence gating] Predictions below 0.6 confidence are suppressed, so uncertain frames never fire stray commands.
• [Temporal smoothing] A 5-clip voting window stabilizes the live output against flicker between similar gestures.
• [Gesture-to-command mapping] A configurable table maps each gesture to a media-player action (next/previous track, volume, play/pause, fullscreen).
• [Dataset explorer view] Jester statistics and the 8-class mapping table inside the demo app.
• [Full evaluation logging] Clip-level accuracy, per-class F1 and the confusion matrix on the Jester validation split — computed during the build, never pre-claimed.

• Complete source code (frame extraction, 3D-CNN training, evaluation, live inference, demo app)
• Jupyter training and evaluation notebook (buyer-run procedure: extract frames, train, evaluate)
• Project report PDF (background, 3D-CNN theory, Jester analysis, methodology, evaluation, error analysis)
• PPT presentation for final review
• Viva Q&A preparation document (3D convolutions, Jester, smoothing/gating, video augmentation)
• Setup guide (environment, Jester download via official annotations, training, running the recognizer)