In this guide
How Social Robots Read Intent: Gestures, Voice Keywords, and Confirmation
A person rarely talks to a robot the way they talk to a vending machine. They start one sentence, change their mind halfway through, wave the robot over, point somewhere else, or correct themselves with a shake of the head. A social robot has to survive that messiness. This guide explains how the current generation of social-robot research approaches the problem, and how a student can build an honest small-scale version of it without pretending to have built a thinking machine.
The headline behind this guide arrived in late October 2026, when Krafton subsidiary Ludo Robotics presented Ludi 0.2 at the Bay Area Robotics Symposium at UC Berkeley on 30 October 2026. Ludi is described by the company as robot AI for social intelligence: reading human intent, gestures, feedback and social cues, keeping context when tasks change mid-stream, and linking movement with interaction. The reported example is a robot heading to fetch a drink that switches to another beverage when the person changes the instruction mid-task. Those are company-reported improvements only. No independent benchmarks were disclosed, and the planned successor, Ludi 1.0, is described as a future foundation model that would fold perception, dialogue and physical action into one system. Treat the headline as a research direction, not a measured result.
What "reading intent" actually means in a robot
Intent, in robotics, is not mind reading. It is a guess about what a person wants, made from observable signals, and a good system is honest about how uncertain that guess is. A wave could mean hello, come here, or stop, depending on context. A point could select an object, a direction, or a place. A spoken phrase could be a new command, a correction of the last one, or background chatter that was never meant for the robot at all.
Research systems attack this with several layers. Perception turns camera and microphone data into features: where the hands are, where the person is looking, what words were spoken. Interpretation maps those features onto likely goals, often with a model that has seen many examples of people interacting. Dialogue and memory keep track of the current task, so a correction can refer back to it. Action turns the chosen goal into movement, and feedback — a face, a voice line, a light — tells the person what the robot understood, so they can correct it before anything goes wrong.
The student-scale version of the same idea is deliberately smaller. Instead of a model that understands free speech and unseen gestures, a student build uses a fixed vocabulary: five hand shapes and six spoken commands, each mapped by a rule to a task. That is not understanding, and a good report says so. But it is still a genuine demonstration of the core loop that matters: sense, map, confirm, act, and remember what changed.
How a robot reads a gesture
A camera sees a hand as pixels. A hand-tracking tool turns those pixels into landmarks: a set of points for the wrist, the knuckles, and the fingertips, usually 21 points per hand. From those points, simple geometry answers the questions a gesture vocabulary needs. Are the fingers extended or folded? Is the palm facing the camera? Is the index finger pointing, and in which direction? Is the hand moving in a wave pattern over the last few frames?
Each gesture in a fixed vocabulary is a rule over those measurements. An open palm is all fingers extended. A thumbs-up is the thumb extended with the other fingers folded. A point is the index finger extended with the rest folded. The rules need thresholds, and thresholds need testing: the same hand looks different in dim light, at an angle, or against a cluttered background. That is why a gesture demo belongs in good light, with a clear view of the hand, and why a report should state the conditions under which each gesture was tested.
Gestures also need time. A single frame can misfire, so a robust build waits for the same gesture to persist across several frames before acting on it, and it tracks the hand across frames so a wave is recognised as a movement pattern, not a single pose. When no gesture matches, the correct behaviour is to ask, not to guess. A robot that acts on a misread gesture will drive to the wrong place with great confidence, which is worse than a robot that visibly asks for clarification.
How a robot hears a keyword without the internet
Offline keyword spotting works differently from a voice assistant in the cloud. A small speech model runs on the device itself and listens continuously for a short list of phrases, such as go station A, stop, or follow me. It does not transcribe everything you say and it does not understand sentences it was never trained to spot. It compares incoming audio against the patterns of its keyword list and fires when the match is strong enough.
That limitation is also the point. A keyword spotter needs no internet connection, keeps audio on the device, and runs on a small board. Its weaknesses are predictable: background noise, similar-sounding words, and accents or speeds it was not tested with will all lower its accuracy. A student build should therefore keep the vocabulary short, test each command in a quiet room first, and report honestly which commands needed a repeat. A spoken confirmation line from the robot — repeating back what it heard — turns a recognition error into something the person can catch and correct.
Combining two channels makes the system stronger than either alone. A gesture and a voice command can mean the same thing, so a person can interrupt the robot by waving or by speaking, whichever is easier in the moment. When the two channels disagree, or when either one is uncertain, the robot should fall back to the same answer in both cases: ask for confirmation.
The state machine: the part students underrate
Behind every polite robot is a state machine, and it is the most transferable skill in this whole guide. A state machine is a small set of named states — idle, listening, working, clarifying, stopped — plus rules for moving between them. Only certain transitions are allowed. A robot in the working state that receives a new command does not silently overwrite its job; it moves to clarifying, pauses, and asks. A robot in the stopped state does not resume on its own; it waits for a new command after the path is clear.
This structure is what makes a mid-task switch demonstrable. The machine remembers the previous target while it asks about the new one, and it writes both targets to a log file, with the command source and the confirmation result. An examiner can then read the log and see that context survived the switch, rather than taking the builder's word for it. The log is the evidence. Without it, a task switch is a story; with it, it is a result.
Design the states before writing any movement code. List every input that can arrive in every state, and decide what each one does. What happens if a stop gesture arrives during clarification? What happens if an obstacle appears during a switch? What happens if the person gives a thumbs-down after a confirmation question? Answering those questions on paper, before the robot moves, is most of the engineering in a social robot.
Why confirmation beats guessing
Every input channel in a social robot is uncertain. Cameras misread hands in poor light. Keyword spotters mishear words in noisy rooms. People themselves are ambiguous: a wave can mean several things, and a correction can arrive before the robot has finished the previous sentence. Acting immediately on an uncertain reading feels responsive, but it manufactures confident errors.
Confirmation trades one gesture for correctness. The robot pauses, turns its head toward the person, shows a questioning expression on its display, and speaks the proposed change aloud. A thumbs-up confirms, a thumbs-down keeps the old target and asks again. The cost is a second of the person's time. The benefit is that errors become visible and reversible, and the demonstration gains a moment the audience can actually see: the robot checking itself before it acts.
This is also why a social robot shows its state. A display face, a coloured light ring, and a spoken line are not decoration. They are the robot's way of saying what it is doing and what it understood, in a form a person can correct. A silent robot that moves confidently is hard to trust; a robot that narrates its own uncertainty is easy to work with.
Building the student version: a practical path
A realistic student build is a tabletop wheeled robot, not a humanoid, with a pan-tilt head carrying a camera, a small display face, a microphone, and a speaker. A Raspberry Pi runs hand tracking and keyword spotting, a motor driver runs two wheels, and an ultrasonic sensor halts the base when an obstacle is close. Three marked floor points — Station A, Station B, and Home — give the robot somewhere to go, and three tasks — go to a station, follow a person, and patrol a loop — give it something to switch between.
The signature demonstration writes itself. Start the robot toward Station A. Interrupt it mid-route by waving or saying go station B. Watch it pause, ask for confirmation, and re-plan only after a thumbs-up, then read the log row that shows the previous target, the new target, the command source, and the confirmation result. Then demonstrate the failure modes on purpose: a thumbs-down that keeps the old target, an unmatched gesture that triggers a clarification prompt, and an obstacle that halts the base in any state. A demo that shows its own limits is more convincing than a demo that hides them.
For the companion student build that follows exactly this design, see the Ludi-inspired social robot project: https://projectech.in/projects/ludi-inspired-social-robot-gesture-voice-intent-task-switching/ — a tabletop wheeled robot with five gestures, six voice commands, a confirmation step before every re-plan, and a CSV task log. The same project page also documents the limits plainly: no arm, no carrying, no free-speech understanding, and follow-me tracking that follows the nearest person rather than identifying anyone.
FAQ
Does a student social robot really understand people?
No. A student build maps a fixed gesture and voice vocabulary to tasks by rules, which is not comprehension of free speech or unseen gestures. The honest contribution is the loop around that mapping: sensing, confirmation, re-planning, and logging, all visible to an examiner. Claiming understanding would be the one mistake that undermines the rest of the build.
Why use both gestures and voice instead of one channel?
Because people switch channels naturally, and because each channel fails differently. A gesture works when the room is noisy; a voice command works when the hands are busy or out of view. When both channels map to the same tasks, the person can interrupt the robot by whichever means is at hand, and disagreement between channels is a signal to ask, not to act.
How many gestures and commands should a student vocabulary have?
Fewer than you think. Five gestures and six commands are enough to demonstrate interruption, confirmation, correction, and ambiguity, and each one can be tested properly. A longer vocabulary that is half-tested will fail in the demonstration. Add a new command only when you can test it in the same conditions as the rest.
What makes a task-switching demo convincing to an examiner?
Evidence, not smoothness. The companion student build shows the pattern in practice: https://projectech.in/projects/ludi-inspired-social-robot-gesture-voice-intent-task-switching/ logs every switch with its previous target, new target, command source, and confirmation result. A log file showing the previous target, the new target, the command source, and the confirmation result proves the switch happened the way you claim. A demonstration of a correction — a thumbs-down keeping the old target — proves the confirmation step is real. A demonstration of an unmatched input producing a question proves the robot does not guess.
What is the difference between this and a remote-controlled robot?
A remote-controlled robot moves when told and has no state of its own. A social robot keeps a task, a target, and a history; it can be interrupted, it asks before changing its mind, and it can say what it understood. The state machine and the log are what separate the two, even when the hardware looks similar.
Can this kind of robot help in a home or a ward?
Not in this student form. A tabletop prototype demonstrates interaction and task switching; it is not assistance, supervision, therapy, or safety equipment for any person. Real deployment would need far more testing, safety engineering, and evidence than a college prototype can provide, and the report should say so.
Limitations
Fixed vocabularies are the central limit: anything outside the gesture and voice lists produces a clarification prompt, not an action, and that is by design. Cameras need good light and a clear view of the hand, and keyword spotters degrade in noise, so demonstrations belong in controlled indoor conditions. Follow-me tracking that follows the nearest detected person cannot identify individuals and can latch onto the wrong person. Small wheels, a capped speed, and an ultrasonic stop suit a flat indoor floor, not outdoors or uneven ground. And the headline research behind this guide is company-reported work with no independent benchmarks disclosed, so every comparison between a student build and Ludi should be framed as inspiration, not as a measured gap closed.
The value of the student version is not that it matches the headline system. It is that it makes the headline system's core problem visible at a scale a student can build, test, and defend: a person changes their mind mid-task, and the robot copes politely, visibly, and with a log to prove it.