Ludi-Inspired Social Robot: Gesture and Voice Control with Intent-Aware Task Switching

A person rarely gives a robot one clean instruction and then stays silent. In a home or a small ward, someone changes their mind mid-sentence, waves the robot over, corrects themselves, or points somewhere else while the robot is already moving, and most student robots cannot cope with that at all. They run one fixed routine: drive to a point, follow a line, or stop at an obstacle. When the instruction changes halfway through, they either ignore the change, finish the old job first, or need a full restart. That gap between a single-command demo and a robot that can be interrupted politely is exactly what social-robotics research is trying to close, and it is a gap a student can demonstrate honestly at tabletop scale without pretending to have built a thinking machine.

The idea surfaced again in late October 2026, when Krafton subsidiary Ludo Robotics unveiled Ludi 0.2 at the Bay Area Robotics Symposium at UC Berkeley on 30 October 2026. Ludi is described by the company as robot AI for social intelligence: reading human intent, gestures, feedback and social cues, keeping context when tasks change mid-stream, and linking movement with interaction. The reported example is a robot heading to fetch a drink that switches to another beverage when the person changes the instruction mid-task. Those are company-reported improvements only: no independent benchmarks were disclosed, and Ludi 1.0 is a planned future foundation model, not a build a student can reproduce. This project translates the headline honestly: a small wheeled robot that accepts a mid-task change, asks for confirmation, re-plans, and logs the switch, using fixed rules rather than a language model.

The end-to-end mechanism runs from a person gesturing or speaking to a robot that pauses, asks, re-plans, and logs the change. The Pi camera watches for hands while the microphone listens for keywords, and both inputs feed the same fixed-vocabulary mapper, so a gesture and a voice command mean the same thing regardless of which channel delivered them. The state machine decides what happens next: a new command during a task moves the robot from working to clarifying, the head turns toward the person, the OLED face shows a questioning expression, and the speaker asks for confirmation. A thumbs-up confirms the switch and the robot re-plans toward the new station while writing both targets to the log; a thumbs-down keeps the old target and asks again. An obstacle inside the stop zone halts the base in any state, and an input that matches nothing in either vocabulary triggers a clarification prompt instead of a guessed action.

1. The builder assembles the two-wheel base, mounts the pan-tilt head with its camera, OLED face, LED ring, microphone, and speaker, fits the HC-SR04 sensor at the front, and wires the drive motors through the TB6612FNG driver to the Raspberry Pi.
2. The builder calibrates the system: each of the five gestures is tested in good light, each of the six voice commands is tested in a quiet room, the three floor points Station A, Station B, and Home are marked, and the obstacle stop is checked by placing an object at 30 cm.
3. For the idle demonstration, the builder starts the robot at Home, shows the listening face and LED state, and waves to get its attention so the head turns toward the person.
4. For the go-to-station demonstration, the builder says go station A or points at Station A, and watches the robot drive toward it while the face shows the working expression.
5. For the signature switch demonstration, the builder interrupts the robot en route to Station A by waving or saying go station B, watches it pause and ask for confirmation on the OLED face and speaker, confirms with a thumbs-up, and watches it re-plan toward Station B.
6. For the correction and ambiguity demonstration, the builder gives a thumbs-down after a switch request to show the old target is kept, then gives an unmatched gesture or unclear word to show the robot asking for clarification instead of guessing.
7. The builder exports the CSV task log, points to the rows showing the previous target, the new target, the command source, and the confirmation result, and reports honestly in the report which commands were recognised first time and which needed a repeat.

Tech stack:

• Raspberry Pi 5 · quad-core host and task state machine
• Pi Camera Module 3 · hand tracking camera input
• MediaPipe hand tracking · on-device gesture landmark detection
• Vosk offline keyword spotting · USB microphone voice commands
• SG90 pan-tilt head · 2 servos plus OLED face and speaker
• LED ring · coloured state and attention signalling
• HC-SR04 obstacle stop · 30 cm safety halt
• TB6612FNG differential drive · 2 DC gear motors and wheels

This robot combines four capabilities that make a complete social-interaction prototype rather than a remote-controlled car with a camera on top. First, two input channels work side by side: a fixed gesture vocabulary read from the camera and a fixed voice vocabulary spotted offline, so a person can interrupt the robot by waving, pointing, or speaking without touching it. Second, task switching is a designed behaviour, not an accident: a mid-task change pauses the base, triggers a spoken and on-screen confirmation, and only then re-plans, so the robot never silently abandons one job for another. Third, the robot shows its state instead of hiding it: the pan-tilt head turns toward the person, the OLED face changes expression, and the LED ring signals listening, working, clarifying, or stopped. Fourth, every switch is logged with its previous target, new target, command source, and confirmation result, so the claim that context survives the switch is backed by a file the examiner can read.

• [Fixed gesture vocabulary] Five hand gestures read by MediaPipe on the Pi: open palm means stop, wave means get attention or greet, point means select a station, thumbs-up means confirm, and thumbs-down means correction — implemented in the base build.
• [Offline voice commands] Six voice commands spotted on-device by Vosk with a USB microphone: go station A, go station B, stop, follow me, patrol, and help, with no internet connection needed for recognition — implemented in the base build.
• [Rule-based task switching] A state machine with idle, listening, working, clarifying, and stopped states accepts a new command mid-task, pauses the base, and re-plans only after confirmation, so a change of instruction never silently overwrites the current job — implemented in the base build.
• [Confirmation before re-planning] When a mid-task change arrives, the OLED face and a spoken line ask the person to confirm, and a thumbs-up confirms while a thumbs-down triggers a correction prompt, so ambiguous input produces a question rather than a guess — implemented in the base build.
• [Context task log with CSV export] Every task, command source, previous target, new target, and confirmation result is written to a CSV log, so the demonstration can show that the robot remembered the old target while switching to the new one — implemented in the base build.
• [Expressive pan-tilt head and face] Two SG90 servos aim the head at the person, a 128 by 64 pixel OLED face changes expression with the robot state, and an LED ring signals listening, working, clarifying, or stopped at a glance — implemented in the base build.
• [Obstacle stop and capped indoor speed] An HC-SR04 sensor halts the base whenever an obstacle is at 30 cm or less, and the drive speed is capped for indoor demonstration, so an interrupted robot stops safely instead of driving on — implemented in the base build.
• [Face-based person identification] Future scope only, not implemented: recognising a specific person by face so Follow-Me tracks one individual in a crowd, listed as an extension rather than a claimed feature, since the base build tracks the centre of the nearest detected person and cannot identify anyone.

• Complete fabricated tabletop social robot (two-wheel base, pan-tilt head, Pi Camera Module 3, OLED face, LED ring, USB microphone, speaker, HC-SR04 sensor, TB6612FNG driver, Raspberry Pi board)
• Gesture and voice control software (MediaPipe hand tracking, Vosk offline keyword spotting, task state machine, confirmation prompts, CSV task logging, control dashboard)
• Setup and calibration guide (gesture testing procedure, voice testing procedure, station marking, obstacle-stop check, task-log reading worksheet)
• Project report PDF (social robotics background, fixed-vocabulary design, state machine design, task-switching demonstration results, honest recognition observations)
• PPT presentation for final review
• Viva Q&A preparation document (social cues versus rule-based mapping, gesture and voice vocabularies, state machines, confirmation design, Ludi comparison limits, follow-me limits)