Gaze Tracking using Webcam and Dlib

Where someone looks is rich information — for attention studies, hands-free interface experiments, usability testing — but commercial eye trackers are specialized hardware. A laptop webcam plus classical computer vision gets surprisingly far: the geometry of the eye is stable, the pupil is the darkest blob in the eye region, and a short calibration turns pupil positions into screen coordinates. This project builds that pipeline with dlib's 68-point landmark model (Kazemi and Sullivan, 2014), pupil-center estimation by image moments, and a 9-point calibration grid that fits a polynomial mapping per user.

Because every face, webcam and lighting setup differs, accuracy is personal: the included notebook runs a calibration procedure the buyer performs, then computes mean angular error over the calibration points, so the report's numbers come from the buyer's own run. The design target is a few degrees of mean angular error at typical laptop distance in good lighting — and the limitations section is blunt about when it breaks: poor light, glasses glare, head movement after calibration. MPIIGaze (Zhang et al., 2017) is cited as the deep-learning background reference for what appearance-based gaze estimation looks like at research scale.

1. The webcam frame is passed to dlib's face detector and the 68-point landmark predictor.
2. Eye regions are cropped from landmarks 36-47 and converted to grayscale.
3. The pupil blob is isolated by adaptive thresholding and its center found with image moments.
4. During calibration, the user looks at nine on-screen targets while pupil positions are recorded.
5. A polynomial mapping is fit from pupil-center vectors to screen coordinates for that user.
6. In tracking mode, each frame's pupil centers are mapped through the calibration to a gaze point, drawn live on screen.

Tech stack:

• Python 3.10, dlib (face detection, 68-point landmarks)
• OpenCV (frame capture, eye cropping, thresholding)
• NumPy (pupil geometry and mapping math)
• scikit-learn (polynomial calibration fit)
• Jupyter notebook (calibration procedure and error analysis)
• Matplotlib (calibration grid and residual plots)
• Tkinter overlay app (calibration targets and live gaze point)

• Dlib 68-point facial landmark detection for precise eye-region localization
• Pupil-center estimation from thresholded pupil blobs via image moments
• 9-point on-screen calibration procedure with per-user mapping fit
• Real-time gaze-point overlay on the screen during tracking
• Gaze heatmap logging: dwell points recorded over a session
• Calibration-quality report with per-point residuals
• Evaluation notebook computing mean angular error on the buyer's own calibration run
• Head-pose gating that pauses tracking when the head moves too far

• Complete source code (landmarks, pupil estimation, calibration, tracking overlay)
• Jupyter calibration and evaluation notebook (procedure, angular-error computation, residual plots)
• Project report PDF (background, eye geometry, calibration method, buyer-run evaluation, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (dlib landmarks, pupil detection, calibration math, MPIIGaze context)
• Setup guide (environment, dlib model download, webcam setup, running calibration)

• Lighting is the dominant failure mode: pupil thresholding degrades in dim or strongly backlit rooms, and the report documents the lighting conditions the evaluation was run under.
• The calibration is valid only for the head position held during calibration; leaning in, turning or slouching afterward shifts the mapping until recalibration.
• Glasses glare, heavy eyeliner and drooping eyelids can occlude the pupil blob and break center estimation for those users.
• This is an educational attention-tracking prototype, not a medical or assistive device — it must not be used for diagnosis, accessibility-critical control or any safety-relevant purpose.