The problem
A gesture-controlled media controller built on the APDS-9960 gesture sensor and the ESP32 lets you run a PC's music or video player with mid-air hand swipes — no touching the keyboard, mouse or phone. Swipe right to skip a track, left to go back, up or down for volume, and push toward the sensor to toggle play/pause. The problem it answers is ordinary: keyboards disappear under papers, hands are wet in the kitchen or greasy in the lab, and reaching across the desk breaks the flow of work. Camera-based alternatives need a webcam and eat CPU; commercial remotes add cost and batteries. The APDS-9960 answers this in hardware: four directional IR photodiodes watch the reflected glow of its own 940 nm IR LED, and the chip compares which pair lights up first to decide the swipe direction. The ESP32 reads the result over I2C and re-emits it as Bluetooth HID media keys, so the PC obeys with zero installed software.
Up, down, left, right, near or far above the sensor
Follow the working sequence
How it works
A hand above the APDS-9960 breaks the symmetry of reflected IR light; the sensor's gesture engine turns that asymmetry into a direction event, and the ESP32 re-emits it to the PC as a media key. The sensor fires its 940 nm IR LED and samples four directional photodiodes that see the hand's reflection in sequence. First-principles view: the chip compares which pair lights up first and in what order, reducing that sequence to one of six events — up, down, left, right, near or far — with no camera and no image processing. The INT pin wakes the ESP32, which reads the event over I2C (address 0x39), applies a 10 ms debounce so a swipe cannot retrigger, and emits the matching HID media-key report over Bluetooth. The optional USB-serial path sends the same event code to a Python helper that injects the key press instead.
- The APDS-9960's proximity engine watches for a hand entering its IR field; when proximity crosses the entry threshold, the gesture engine activates automatically.
- The on-chip 940 nm IR LED pulses and the four directional photodiodes sample the reflected glow in sequence, storing the datasets in the 32-byte FIFO.
- The gesture engine compares the order in which the U/D/L/R diodes peak and emits one event — UP, DOWN, LEFT, RIGHT, NEAR or FAR — pulling the INT pin low.
- The ESP32 wakes on the interrupt, reads the event over I2C (address 0x39), and runs a 10 ms debounce on the gesture FIFO so one swipe is reported exactly once.
- The firmware looks the event up in a mapping table (default: RIGHT → next track, LEFT → previous, UP/DOWN → volume, NEAR → play/pause).
- In BLE HID mode the ESP32 transmits the matching consumer-control key report; the paired PC executes it in the active media player with no installed software.
- In USB-serial mode the same mapped command goes out over USB serial at 115200 baud, where the Python helper converts it into a media key press via pyautogui.
Tech stack:
- ESP32 DevKit (Bluetooth HID keyboard + USB serial)
- APDS-9960 gesture sensor (I2C, address 0x39)
- Arduino IDE · ESP32 core · C/C++
- SparkFun APDS-9960 library (gesture engine API)
- BLE HID consumer-control profile
- Python 3 · pyautogui (USB-serial mode only)
- 3.3 V logic-level design (the sensor is not 5 V tolerant)
- Breadboard prototype · jumper wiring
| Parameter | Value |
|---|---|
| Gesture detection range | Approximately 10–20 cm (4–8 inches, APDS-9960 datasheet) |
| Reliable working zone | Approximately 5–15 cm above the sensor (design target) |
| Gesture events | 6: up, down, left, right, near, far (on-chip engine) |
| Gesture-to-keypress latency | Approximately 120 ms (design target, not a measured claim) |
| Sensor supply voltage | 3.3 V (operating range 2.4–3.6 V per datasheet — not 5 V tolerant) |
| I2C interface | Fast mode up to 400 kHz, address 0x39 (datasheet) |
| Gesture FIFO | 32 datasets (datasheet) |
| Wireless link | Bluetooth HID consumer control; pairing range approximately 10 m (design target) |
| Controller | ESP32 dual-core at 240 MHz, 4 MB flash (datasheet) |
| USB-serial fallback | 115200 baud to the Python helper script |
| Power | 5 V USB; expected operating current approximately 120–180 mA during gesture bursts |
Project features
The four capabilities that carry the build are gesture decoding, the HID bridge, the dual-link option, and tuneable sensitivity. Gesture decoding runs on the APDS-9960 itself: its four photodiodes (Up, Down, Left, Right) measure reflected IR in sequence, and the on-chip engine reduces a swipe to a direction event — the ESP32 never processes raw images, so there is no camera and no image pipeline to maintain. The HID bridge is what makes the PC obey: the ESP32 advertises as a Bluetooth keyboard and sends standard media-key reports (play/pause, next, previous, volume up/down), which every major OS and player understands natively. The dual-link option adds a USB-serial mode, where the same gestures reach a small Python helper on the PC for computers without Bluetooth. Finally, sensitivity is tuneable: gesture threshold, FIFO debounce timing and LED drive current are all parameters, so the controller can be tightened for a desk demo or relaxed for longer reach.
- [On-chip gesture decoding] The APDS-9960's four directional IR photodiodes track the reflected 940 nm LED glow; its gesture engine emits up/down/left/right/near/far events over I2C, so no camera or image processing is needed.
- [BLE HID media keys] The ESP32 pairs as a Bluetooth keyboard and sends standard HID consumer-control reports — play/pause, next, previous, volume up/down — so any media player works with no PC-side software.
- [USB-serial fallback mode] On PCs without Bluetooth, the ESP32 streams gesture codes over USB serial to a small Python helper that issues the same media keys via pyautogui.
- [Interrupt-driven firmware] The sensor's INT pin wakes the ESP32 only when a gesture event is ready, keeping the controller in light sleep between events and the main loop clean.
- [Configurable gesture mapping] Which swipe does what (right → next track, left → previous, up/down → volume, near → play/pause) lives in a documented header file, so the mapping can be rearranged without touching the firmware logic.
- [Tuneable sensitivity] Gesture entry threshold, FIFO debounce timing and IR LED drive current are parameters tuned during the bench test and recorded in the build notes.
- Optional add-on (not part of the base build): phone pairing over BLE HID for Android/iOS media apps — offered separately, untested in the base version.
What is included
- Assembled and tested breadboard prototype (ESP32 DevKit + APDS-9960 breakout, wired)
- Complete firmware source (Arduino sketch, BLE HID mode + USB-serial mode) with wiring notes
- Documented gesture-mapping header with the default swipe-to-command assignments
- PC-side Python helper script for USB-serial mode
- Project report PDF (background, sensor theory, firmware, build notes, results)
- PPT presentation for the final review
- Viva Q&A preparation document (gesture physics, I2C, BLE HID, debouncing)
Limitations & prerequisites
- Short range only: gestures work at 5–15 cm; beyond about 20 cm the sensor stops seeing the hand (datasheet limit) — no long-range or across-the-room control.
- Six events is the whole vocabulary — no multi-finger tracking, no hand-shape recognition, no through-body sensing.
- Direct overhead tube-light can false-trigger the proximity stage; the build mitigates it with a 10 ms FIFO debounce and small side shields, documented in the report as a bench note.
- BLE HID mode needs a Bluetooth-capable PC; USB-serial mode needs Python and the pyautogui package installed on the PC.
- Gestures must be deliberate swipes — very slow drifts or very fast flicks can be missed or misread by the on-chip engine.
- The prototype is USB-powered; battery operation and deep-sleep current figures are not part of the build.
Frequently Asked Questions
What gestures does it detect?
The APDS-9960 reports six events: swipe up, down, left and right, plus near (hand approaching) and far (hand pulling away). This build maps them to media commands — right for next track, left for previous, up/down for volume, near for play/pause. That is the complete vocabulary: it cannot track multiple fingers or recognise hand shapes.
Does the PC need any software installed?
No, in BLE HID mode. The ESP32 pairs as a Bluetooth keyboard and sends standard media-key reports, so VLC, Spotify, YouTube and OS media controls respond with no driver or app installed. The optional USB-serial mode uses a tiny Python helper that turns serial commands into key presses — that one does need Python and the pyautogui package on the PC.
How far from the sensor do gestures work?
Gestures register reliably at about 5–15 cm above the sensor, inside the 10–20 cm (4–8 inch) detection range given in the APDS-9960 datasheet. Closer than about 3 cm the sensor saturates; beyond about 20 cm it stops seeing the hand. The sensor faces upward from the breadboard, so the working zone is a comfortable hover above the desk.
Will room lighting interfere?
The APDS-9960 uses its own 940 nm IR LED and subtracts ambient light, so normal room light is rejected. Builder note: direct overhead tube-light can still cause false triggers — the build fixes it with a 10 ms debounce on the gesture FIFO plus small side shields that block stray light without covering the sensor window.
Can it control a phone instead of a PC?
Not as shipped. The firmware implements a BLE HID keyboard, which phones accept, but gesture-to-media behaviour on Android/iOS varies by app and is untested in this build. The delivered version is validated against a PC or laptop; phone support is listed as future scope in the report.
Is this project suitable for a final-year project?
Yes — it covers embedded sensing, I2C communication, interrupt-driven firmware, wireless HID and a real end-to-end application. The working prototype plus report, PPT and viva Q&A cover gesture physics, the BLE stack and debouncing, giving the student plenty of defensible technical depth for the review.
Components & software requirements
- ESP32 DevKit (Bluetooth HID keyboard + USB serial)
- APDS-9960 gesture sensor (I2C, address 0x39)
- Arduino IDE · ESP32 core · C/C++
- SparkFun APDS-9960 library (gesture engine API)
- BLE HID consumer-control profile
- Python 3 · pyautogui (USB-serial mode only)
- 3.3 V logic-level design (the sensor is not 5 V tolerant)
- Breadboard prototype · jumper wiring
Delivery information
Built-to-order project. Delivery timeline is shared after order confirmation based on current queue.
Support terms
Complete documentation, setup guide, and viva preparation included. Support for setup and explanation provided.
Download free synopsis (PDF)
Enter your name and WhatsApp number — the synopsis PDF downloads immediately. We may send occasional project updates on WhatsApp; no spam.