Motion tracking: Kimodo, retargeting and BeyondMimic
Instead of writing a new reward for every skill, Marvin Bot learns one policy (a trained neural-network controller, page 16) that copies a reference motion. A new skill is then a new clip. This page follows a clip from a text prompt to a trained tracker, and lets you break the two steps that most often go wrong: scaling a human to a 1.41 m robot, and the shape of the reward.
Think of a dance class: rather than explaining in words what a good bow looks like, the teacher does it and you copy. A humanoid is expected to do many things besides walking, such as waving, bowing, squatting or clapping, and writing a separate scoring rule for each would take weeks per skill. With motion tracking the robot is shown a recording and scored on how closely it copies it. The hard parts are getting a good recording for a body that is not human-shaped, and deciding what “close enough” means.
The idea in one picture
Section titled “The idea in one picture”For walking (page 16) the reward is hand-written: follow the commanded velocity, keep the body upright, do not waste torque, lift the feet. That works for one skill, but every new skill (a bow, a squat, a wave) needs new reward terms and weeks of tuning. Motion tracking turns the problem around: give the policy a reference motion, a recording of where every body part should be at every moment, and reward it for staying close. The reward is the same for every skill; only the clip changes.[1]
Words in the pipeline: Kimodo is a diffusion model, a neural network that generates new motions (much as image generators make pictures from text). SOMA is the human skeleton it outputs; 30 fps means 30 poses (frames) per second. A root path is the route the hips should follow; keypoints are tracked points on the body such as wrists, knees and ankles.
Two of these steps hide most of the difficulty. Retargeting has to turn a 1.75 m person into a 1.41 m robot with different proportions and joint limits, and every compromise it makes becomes part of the “truth” the policy is trained to copy. The tracker’s reward has to say how close is close enough. The two playgrounds below let you try both.
Playground A: scaling a person to Marvin Bot
Section titled “Playground A: scaling a person to Marvin Bot”The left panel shows a human source (a stick figure with typical adult proportions). The right panel shows Marvin Bot’s skeleton (lengths from description/robot.yaml and skeleton.yaml) doing its best to put its joints on the scaled human keypoints (hollow circles). Legs are solved exactly (two-link inverse kinematics: working backwards from where the foot should be to the joint angles that put it there), arms by a small optimisation that nudges four joint angles until the hand gets as close as it can, the same kind of solver PyRoki uses. Joint limits are applied after the solve; any limit the motion asks the robot to break is listed in red.
Start here: leave Reach forward playing, press Uniform by height, then Marvin Bot today, and watch the feet and the readouts. How to read the picture: the robot (orange) is trying to put its joints on the hollow blue circles (the shrunk human, shown in grey on the left at full size); red marks show where it misses or hits a joint limit. Good readouts: hand error a few centimetres, stance sole height near 0 mm (on the floor), little foot sliding, a standing knee bend near the robot’s 15° stance, and no joint-limit hits. The lower chart plots the shoulder over the clip: the purple path should stay out of the shaded (forbidden) area.
From first principles
Section titled “From first principles”Why a reference motion?
Section titled “Why a reference motion?”Reinforcement learning (page 16) needs a number, the reward, that says how good each moment was. For walking, people write it by hand. A tracker’s reward is generic: “be where the reference is”. BeyondMimic showed that one such formulation, with one set of hyperparameters, learns sprints, cartwheels and dances and transfers to a real humanoid without per-skill tuning.[1] The price is that you need good reference motions, and that the robot must be physically able to do them.
Step 1: making motions from text (Kimodo)
Section titled “Step 1: making motions from text (Kimodo)”Kimodo is NVIDIA’s kinematic motion diffusion model, trained on about 700 hours of commercially usable optical motion capture. It takes a text prompt and optional constraints (full-body keyframes, hand or foot positions, a 2D root path or waypoints) and outputs a human motion on NVIDIA’s SOMA skeleton.[2][5]
Go deeper: Kimodo’s licences
Its code is Apache-2.0; the SOMA “RP” model is under the NVIDIA Open Model License; its text encoder (the part that turns the prompt into numbers) is Meta’s Llama 3 8B Instruct (gated, Meta Llama 3 Community License) with LLM2Vec adapters.
“Kinematic” means it outputs positions and angles only, not forces: nothing guarantees the motion is physically possible, which is why the clips are checked.
Wording alone does not set speed. In the repo, “a person jogs forward” produced a 2 m/s run that no tracker could follow. So every locomotion clip and in-place skill gets a root path: the hips’ speed, heading and direction of travel frame by frame (generate_kimodo.py). Several samples are generated per clip (4 by default, 8 for the clips that needed it) and the one with the fewest failed skill checks, then the least foot sliding and crossing of the feet, is kept.
Step 2: retargeting (PyRoki)
Section titled “Step 2: retargeting (PyRoki)”The robot has different segment lengths, fewer joints (no waist pitch, no wrist roll) and joint limits. Retargeting finds, frame by frame, the robot joint angles q that best put the robot’s body points on the human’s keypoints (pelvis, hips, knees, ankles, toes, shoulders, elbows, wrists). PyRoki writes this as a least-squares problem (add up the squared misses and make the total as small as possible) and solves it with a Levenberg–Marquardt optimiser, a standard method that repeatedly nudges the angles downhill, written in the JAX numerical library:[3]
minimise over q: Σk wk ‖FKk(q) − s ⊙ pk‖² + wlim·limits(q) + wcoll·self-collision(q) + wrest‖q − qrest‖² + smoothness
Here FKk(q) is forward kinematics: the position of robot body point k for joint angles q; pk is the human keypoint; s is the scale (⊙ means each direction can be scaled separately); the w are weights saying how much each wish matters; ‖…‖² is the squared distance. In words: put the robot’s points on the shrunk human’s points, stay inside the joint limits, do not let limbs pass through each other, stay near a resting pose, and move smoothly.
The scale is the hard part. Shrink everything by the height ratio (1.438/1.75 = 0.82) and the human’s leg is still 0.70 m long while Marvin Bot’s straight leg is 0.640 m: either the feet float or the pelvis must drop. Shrink by too much and the robot has to crouch. Arms, torso and legs each need their own scale, which is what the playground’s “Per body part” mode does. GMR, a retargeter built for exactly this problem, also uses non-uniform local scaling followed by a two-stage optimisation, and its authors show that retargeting artefacts (foot sliding, self-penetration) directly hurt the trained tracker.[6]
What the playground’s solver does, exactly
Human proportions come from the repo’s measured source (hip-to-ankle 0.853 m and shoulder-to-wrist 0.56 m for a 1.75 m person) plus standard anthropometry for the rest. Each frame, the human’s joint angles give its keypoints; they are scaled (uniformly about the floor point under the hips, or legs about the floor, shoulders about the hips, arms about each shoulder, as in scale_keypoints()). The robot pelvis is pinned to the scaled pelvis target with the human’s pelvis tilt. Each leg is solved analytically (hip roll, hip pitch, knee) to reach its ankle target, then clamped to its limits; the ankle pitch keeps the foot flat within ±40°. Each arm (shoulder pitch, roll, yaw, elbow) is solved by 30 damped Gauss–Newton steps on wrist error plus 0.3 × elbow error plus a small pull to the human’s angles; it is solved once without limits (to find what the motion asks for) and once with limits, including the coupled shoulder-roll limit. Simplifications: no self-collision, no smoothness term, no root optimisation (PyRoki moves the root too), and a 2-link leg instead of the full chain.
Step 3: checking the clips
Section titled “Step 3: checking the clips”A bad clip wastes hours of training and teaches the wrong thing, so sim/motion/check_clips.py runs four layers, the first three on Kimodo’s human clip and the third again on the retargeted robot clip, so a failure points at generation or at retargeting:
- Text match. NVIDIA’s text–motion retrieval model for SOMA (TMR-SOMA-RP-v1, the model Kimodo’s benchmark scores with) turns every clip and every prompt into a list of numbers (an “embedding”) placed so that a motion and a sentence describing it land close together. Each clip gets its similarity to its own prompt (cosine mapped to 0–1) and that prompt’s rank among all 25. A clip that another prompt describes better is suspect.
- Motion quality. Kimodo’s benchmark metrics: foot sliding from toe height and from Kimodo’s contact labels, the share of contact frames that skate, contact consistency, and how far the hips strayed from the requested root path. Kimodo’s published foot skate is 2–4 cm/s.
- Skill checks. A geometric test per clip, in units of the standing head height H so it runs on both bodies: the hands meet (closer than 0.07 H) in front of the chest at least three times in a clap; a hand above the shoulder 40% of the time and swinging in a wave; the pelvis dropping by 30% in a squat; distance within ±25% and turn within 25° of the root path for a walk; both feet off the ground in 5% of jog frames, and so on.
- Review renders. The human as a stick figure beside the robot reference, with the checks written on the frame.
On the first 25 clips every clip ranked its own prompt first or second and slid 0.2–3.8 cm/s, but five did less than asked (the wave waved for 2 s of 8; the arm raise reached about 45°). Those five were regenerated with sharper prompts and more samples, now picked by the skill checks, and all 25 pass on the human. On the robot two fall short: the clap’s palms meet once rather than three times, and the pick-up’s hands stop 27 cm above the floor, because with no waist pitch the robot cannot bend over.
Step 4: the tracker’s reward (BeyondMimic)
Section titled “Step 4: the tracker’s reward (BeyondMimic)”The tracker observes its own joint angles, joint velocities, IMU (the inertial sensor in the pelvis) and last action, plus the reference: the target pose at the next few frames (ProtoMotions uses 1, 2, 4 and 8 frames ahead) and the error of an anchor body (the torso) against the reference’s anchor.[1][4] Every other body’s target is expressed relative to the anchor, so a small drift in where the robot stands does not count as a pose error. Each tracking term is an exponential kernel, a scoring curve that pays 1 for a perfect match and fades towards 0 as the error grows:
r(ē, σ) = exp(−ē / σ²), ē = mean over bodies of ‖xrobot − xref‖²
ē is a mean squared error (m² for positions, rad² for orientations), and σ (sigma) is the “tolerance”: an error of one σ gives exp(−1) = 0.37 of the reward, two σ gives 0.02. In BeyondMimic σ is 0.3 m for body positions, 0.4 rad for orientations, 1.0 m/s for linear and 3.14 rad/s for angular velocities, each with weight 1.0, plus optional anchor position and orientation terms with weight 0.5.
Penalties are −10 for soft joint limits (0.9 of the mechanical range), −0.1 for action rate and −0.1 for undesired contacts.[1]
An episode (one attempt) ends when the anchor or an end effector (a hand or foot) is 0.25 m too high or low, or the anchor is tilted 0.8 rad from the reference. Training starts episodes at random points of the clips, more often in the one-second bins where the policy has been failing (adaptive sampling, a moving average of failures with a uniform floor).[1]
The kernel’s shape matters. Its slope, dr/de = −(2e/σ²)·r, is zero at e = 0 and largest at e = σ/√2. With σ = 0.3 m, a body 3 cm off keeps exp(−0.03²/0.3²) = exp(−0.01) = 0.990 of its reward and the gradient is tiny. Playground B shows what that does.
Playground B: the reward kernel
Section titled “Playground B: the reward kernel”How strict should the score be? Here you set the tolerance σ and an error, and see how much reward the robot keeps and how strongly the score pushes it to improve (the slope: zero slope means “no reason to fix this”). Start here: press 3 cm crouch and read Reward lost: well under 1%, too little for training to bother fixing. Then press the other scenarios. How to read the chart: the solid blue curve is the reward if every scored body is off by the error on the x-axis; the dashed purple curve is the same when only some of them are off (the average dilutes the miss); the orange dot is your current setting. A flat curve under the dot means the robot feels almost no pull towards the reference.
The weighted sum
Section titled “The weighted sum”The policy sees one number per step: the weighted sum of all terms. Slide the errors and see how much each term contributes. The defaults are Marvin Bot’s tracker (mlp_bm_marvin_bot.py as written by setup_protomotions.py).
How Marvin Bot does it
Section titled “How Marvin Bot does it”What Marvin Bot changed from upstream’s BeyondMimic config in ProtoMotions, and why (each was found by a failed tracker):
- Scale. Legs by leg length, shoulders about the hips (0.62 as wide, about 0.955 as high, 6.5 cm forward), arms about each shoulder; the knees and feet moved 1.5 and 3 cm outwards because the shrunk human feet ended up 12–14 cm apart and Marvin Bot’s feet are 11 cm wide. With the shoulders or pelvis left high, the solver crouched the robot (knees 37–58° through a walk). The solver also starts from the standing pose, not the middle of each joint’s range (a 65° knee).
- Arms kept out of the body. Upstream leaves the self-collision cost off; retargeted hands went 4–10 cm into the thighs in most walking frames and the palms 9 cm through each other in the clap.
retarget_marvin_bot.pyturns it on (weight 10, margin 2 cm) for the arms against the trunk, legs and the other arm. - Soles. ProtoMotions’ converter puts the lowest body origin 4 cm above the floor, which suits another robot’s foot frame.
fix_sole_height.pylifts each clip until the flatter (stance) foot is on the floor; lifting until the single lowest corner touched left walks 3–5 cm in the air, and the trackers hopped to keep up. - Anchor reward and termination. BeyondMimic’s global anchor-position reward (weight 0.5, σ 0.3 m) and a “fell behind” termination at 0.5 m were missing upstream; without them trackers 1–4 stood still.
- Contact penalty −2.0 per foot whose contact differs from the reference’s. Without it the trackers hopped (both feet down 70% of the time, both up 20%); −0.5 only halved it; at −2.0 both feet are up 1–2% of the time.
- Fifteen scored bodies: BeyondMimic’s 14 key bodies plus the head. Scoring all 28 let an arm hang while the reference raised it (hands 40–75 cm off); leaving the head out made it shake (46–60°/s RMS standing still).
- Joint-angle term, weight 0.5, σ 0.3 rad, exp(−mean squared joint error/σ²) over all joints. The body-position kernel paid 99% for a body 3 cm off, so the tracker stood and walked 5–8° deeper in the hips, knees and ankles than the reference, and with an arm overhead chose a different arm posture for the same hand position (shoulder yaw 95° off in the two-hand wave).
- Looser velocity term (σ 0.5 m/s instead of 1.0), more exploration (log std −2.3), softer gains for an action scale of at least 0.4 rad, and an episode end when any scored body is 25 cm from the reference.
- Adaptive sampling in ProtoMotions is per clip: a clip’s sampling weight goes back to 1 when the policy fails it and decays by 0.999 per evaluation when it succeeds (BeyondMimic’s own version works on one-second bins within a clip).
- A second check. ProtoMotions counts a clip as tracked when the torso height stays within 25 cm;
check_tracking.pyalso compares distance covered, joint angles and the hands and feet alone. That is how the first tracker’s shuffling on the spot (0–32% of the reference’s distance) showed up.
The 25 skills
Section titled “The 25 skills”From docs/skills.md (provisional: the tracker before v4, which is still training): joint RMS error against the reference, and the hands’ mean distance from where the reference puts them, both relative to the pelvis. Click a column heading to sort.
Trade-offs and alternatives
Section titled “Trade-offs and alternatives”| Choice | Picked | Alternatives considered | Why |
|---|---|---|---|
| Motion source | Kimodo (text to motion) | 100STYLE motion capture (CC-BY-4.0), CMU database, BONES-SEED, AMASS, LAFAN1, own motion capture | Same eight motions, tracked the same way: 100STYLE kept the tracker up on 5 of 8, Kimodo 5 of 8 with gentler prompts, CMU 3 of the 5 that passed the filter (feet slid 0.43 m/s median vs 0.06–0.07). Kimodo makes skills no open dataset has; BONES-SEED’s licence covers only non-profits and small companies; AMASS and LAFAN1 are non-commercial or no-derivatives. 100STYLE stays as a locomotion option. |
| Retargeter | ProtoMotions’ PyRoki retargeter, with Marvin Bot’s link map, scaling and self-collision | GMR[6] (named in the 2026-10-25 compute decision and in ACKNOWLEDGMENTS.md as a possible alternative) |
PyRoki comes with ProtoMotions’ Kimodo route (SOMA converter and keypoint extraction), so no converter of Marvin Bot’s own is needed. |
| Tracker framework | ProtoMotions on Newton[4] | mjlab’s BeyondMimic task (kept as a fallback), BeyondMimic’s original code | One framework for retargeting, motion library and training. |
| Skills by tracking vs by reward | Tracking for skills beyond walking | Reward per skill | A new skill is a new clip. The command-driven walking and stair policies stay: a tracker needs a reference to follow, and a joystick does not produce one. |
| Not yet tried | – | A style discriminator (AMP), BeyondMimic’s gain rule, a stronger smoothness term | Listed in docs/decisions.md (2026-10-29) as next options for the two-hand wave and the clap. |
Decisions you can make here
Section titled “Decisions you can make here”- How to scale: by height, by limb length, or per body part. Per body part keeps the feet on the floor and the knees straight, at the cost of changing the motion’s proportions.
- Which joint limits to relax (the coupled shoulder roll is already relaxed with the arm forward) and which skills to drop because the body cannot do them (no waist pitch: no picking things off the floor).
- Reward tolerances: tighter σ for precise poses vs looser for faster early learning; whether to add terms (joint angles, contacts) or a style discriminator (a second network that scores whether a motion “looks human”). Each extra term fixes one failure but adds a weight to tune and another hours-long training run to test it.
- How many bodies to score, and whether hands deserve their own term. Fewer bodies make each one count more (no hanging arm) but leave the unscored ones free to do odd things (the shaking head).
Built from your settings above. Paste it into an issue, a design discussion or an AI assistant session.
Further reading
Section titled “Further reading”- BeyondMimic (Liao et al., 2025): read the supplementary material for the exact rewards, terminations, adaptive sampling and the PD-gain rule.
- Retargeting Matters / GMR (Araújo et al., 2025): a careful study of how retargeting artefacts hurt tracking.
- PyRoki (Kim et al., 2025): how kinematic problems become composable costs.
- Kimodo: the prompting advice and constraint types.
- ProtoMotions: the training framework, with BeyondMimic-style experiments.
- SONIC (Luo et al., 2025): what motion tracking looks like scaled to 700 hours of data; see page 18.
Sources
Section titled “Sources”- Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, C. K. Liu, “BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion”, arXiv:2508.08241, 2025 — used for: tracking formulation, observations, anchor body, kernel form, σ values and weights, penalties, terminations, adaptive sampling.
- NVIDIA, Kimodo repository (nv-tlabs/kimodo) — used for: what Kimodo is, constraint types, skeletons, licences of code and models.
- C. M. Kim, B. Yi, H. Choi, Y. Ma, K. Goldberg, A. Kanazawa, “PyRoki: A Modular Toolkit for Robot Kinematic Optimization”, arXiv:2505.03728, 2025 — used for: retargeting as cost minimisation, Levenberg–Marquardt solver in JAX.
- NVIDIA, ProtoMotions repository (NVlabs/ProtoMotions) — used for: BeyondMimic-style experiment config (future frames 1/2/4/8, per-clip motion weights).
- D. Rempe, M. Petrovich, Y. Yuan et al., “Kimodo: Scaling Controllable Human Motion Generation”, arXiv:2603.15546, 2026 — used for: Kimodo as a kinematic motion diffusion model.
- J. P. Araújo, Y. Ze, P. Xu, J. Wu, C. K. Liu, “Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking”, arXiv:2510.02252, 2025 — used for: GMR, non-uniform scaling, effect of retargeting artefacts on tracking.
- I. Mason, S. Starke, T. Komura, 100STYLE dataset (CC-BY-4.0), 2022 — used for: the alternative motion source.