In this guide
How Sim-to-Real Reinforcement Learning Works: Training a Robot to Walk Before It Ever Moves
On 21 October 2026, at NVIDIA's GTC Berlin keynote, four tiny duck-shaped robots sat, stood and napped on command. The robots were Microducks from Pollen Robotics, and the interesting part was not the trick itself — it was where the trick came from. Nobody programmed those movements joint by joint. Each behavior was learned inside a physics simulation and then transferred to the real robot. This guide explains that method — sim-to-real reinforcement learning — the way an engineering student actually needs it: what is simulated, what the reward means, why transfer fails, and how to close the gap honestly.
The core idea in one paragraph
Reinforcement learning (RL) treats movement as a learned mapping from observations to actions. The robot's controller — the policy — reads sensor values and joint angles, and outputs motor commands. In simulation, that policy can fail thousands of times per minute with zero repair bills: the simulated robot falls, gets reset instantly, and tries again. The learning algorithm, usually PPO (Proximal Policy Optimization), adjusts the policy toward actions that earned higher reward. When the policy is good enough in simulation, it is exported — typically as an ONNX file — and run on the real robot. If the simulation was faithful enough, the real robot walks. The whole discipline of sim-to-real is about making "faithful enough" true.
Why simulation first
Training on hardware alone is slow, expensive and destructive. A walking policy needs millions of steps of experience; a real servo biped covers a few steps per second, breaks when it falls badly, and needs a human to pick it up every time. Simulation parallelizes this: Pollen reports training Microduck's policies with 4,096 simulated robots at once, finishing a usable walking gait in roughly one to two hours on a single CUDA GPU. A student cannot casually match that scale, but does not need to — a small biped model with a few dozen parallel environments still trains a walking gait in an evening on a decent laptop GPU, and far slower on CPU alone. The trade is honest: simulation buys you cheap failures, and the price is the reality gap.
The four pieces of a sim-to-real pipeline
1. The model. You build a digital twin of your robot in a physics engine such as MuJoCo: link lengths, masses, joint limits, servo torque limits and ground friction, matched to the printed hardware. Every mismatch here becomes a transfer failure later. If your simulated leg is 2 cm longer than the printed one, the policy learns a gait the real robot cannot reproduce.
2. The reward. The reward function is the only specification the learner gets. A typical walking reward combines forward velocity (move forward), an upright-posture bonus (do not lean), and a penalty on joint effort and jerky motion (do not flail). Get the reward wrong and you get a robot that has learned to cheat — hopping forward on one foot, or shuffling without lifting its feet, because the reward never said "walk properly". Reward design is where most student RL projects actually live or die.
3. The algorithm. PPO is the default choice for a reason: it is stable, well-documented, and implemented in libraries such as Stable-Baselines3 that a student can run in an afternoon. PPO updates the policy in small, clipped steps so one bad batch of experience cannot destroy a working gait. You do not need to invent an algorithm; you need to configure one correctly — learning rate, batch size, and how many steps run between updates.
4. The transfer. The trained policy is exported to ONNX and run on the robot's computer at a fixed loop rate — Microduck runs its policy at 50 Hz, and a student build targets the same rate. The robot reads its IMU and joint state, feeds them through the policy, and writes servo commands. Nothing about the real world is allowed to surprise the loop: sensor noise, timing jitter and servo delay all have to be inside what training prepared for.
The reality gap, and domain randomization
The reality gap is the difference between simulation and hardware: friction is never exactly as modeled, servos have backlash, batteries sag under load, commands arrive late. A policy trained on one perfect simulation shatters on contact with the first real floor. Domain randomization is the standard fix: during training, you randomize the things you cannot model exactly — battery voltage, joint backlash up to about a degree, command delay, floor friction — so the policy learns a gait that works across a range of worlds instead of one perfect one. Microduck's reported recipe randomizes exactly these quantities. Randomization does not eliminate the gap; it buys robustness, and the remaining gap is measured in transfer trials, not wished away.
How transfer trials are actually run
Serious sim-to-real work never claims success from simulation alone. The honest procedure is staged: first, run the policy with the robot supported — an overhead tether or a helper holding a support frame — and compare the real gait against the simulated one. Then run untethered trials on a soft mat and count what matters: distance walked before a fall, falls per minute, and self-right successes over repeated attempts. These numbers, not the reward curve, are the project's results. A reward curve that climbs beautifully while the real robot falls over in three steps is a failed transfer, and saying so is good engineering, not a failed project.
Picking your own behavior to train
Start with walking, because it has the clearest reward and the clearest failure mode. Then add one behavior at a time — sit, stand, crouch — each as its own policy, switched by a dashboard button or a gamepad. Training one policy per behavior is simpler and more debuggable than training one giant policy that does everything, and it mirrors how the reference platforms ship: a library of behaviors, each trained and tested separately. A student build that walks, sits and self-rights reliably beats a build that claims ten behaviors and demonstrates none of them.
What Microduck changed, and what it did not
Microduck matters because it packaged the whole method into a $399 open platform: open training code, an open simulation stack, and behaviors anyone can retrain. It did not remove the hard parts. Its policies still needed careful reward design, still needed domain randomization, and still faced a reality gap between MuJoCo and hardware. What it proved is that the method scales down to a desk-sized robot — which is exactly why a student version with commodity servos is a legitimate project rather than a toy imitation. If you want the full build that applies this guide end to end, see the companion project: Microduck-Inspired Sim-to-Real Biped Robot.
Frequently asked questions
Do I need a GPU to train a walking policy?
For comfortable iteration, yes — a CUDA GPU turns an evening of training into an hour or two. On a CPU-only laptop the same training still works, just much slower, and you should shrink the model and the number of parallel environments accordingly. Document the hardware you actually used; reviewers respect that more than an optimistic claim.
Why PPO and not another algorithm?
PPO is stable and forgiving: its clipped updates stop one bad batch from destroying a working gait, and it is implemented in beginner-friendly libraries. Newer or fancier algorithms can train faster, but they fail in more confusing ways, which is the wrong trade for a first sim-to-real project.
How do I know my simulation is good enough?
You do not know until transfer. The warning signs are visible before that: if the simulated robot's gait looks nothing like a physical gait — feet sliding, joints snapping, impossible torque — the model is wrong, not the learner. Fix link lengths, masses and torque limits first; never tune the reward to compensate for a broken model.
What is the single most common transfer failure?
Unmodeled delay and backlash. The policy expects instant, exact servo response; real servos lag and have play in their gears. Randomizing both during training is the highest-value fix, and it costs nothing but training time.
Can I train directly on the real robot instead?
Only for tiny, safe refinements — and only after a simulated policy already works. Pure real-world training needs thousands of falls, and a servo biped does not survive that. Simulation first, hardware second, is the order that keeps both the robot and the project alive.
Limitations of this guide
This guide explains the method, not a specific robot's numbers: reward values, training times and loop rates all depend on your hardware, and any guide that quotes them as universal is guessing. Domain randomization improves robustness but never guarantees transfer — some policies still fail on hardware, and that failure is data, not defeat. Finally, sim-to-real learning is not a shortcut around mechanics: a badly built biped with loose joints and an unbalanced body will not be saved by any amount of training.
Where to go next
Build the MuJoCo model of a real mechanism before touching any learning code — if the model is wrong, everything downstream is fiction. Train walking first, measure transfer with tethered trials, and only then add behaviors. The companion project linked above follows exactly this order, with the dashboard, the fall test and the CSV episode logs that turn a demo into a reportable experiment.