HuMoTAR
Research / Embodied intelligence

HuMoTAR: Semantic Humanoid Retargeting with Foundational Priors

Narayanan Palghat Parameswaran1 Jay Karhade1 Haoyang He2 Neehar Peri3 Jay Patrikar2 Dong-Ki Kim2 Shayegan Omidshafiei2 Sebastian Scherer1,2 Deva Ramanan1
1Carnegie Mellon University 2FieldAI 3California Institute of Technology
A gallery of recorded humanoid–object interactions

TL;DR

1. From human motion to humanoid interaction

Motion retargeting translates a human demonstration into a reference motion for a robot. Human and robot bodies differ in their proportions, joint limits and ranges of motion. Existing methods typically match the demonstrated body motion while adapting contact targets to the robot’s geometry.

We posit that retargeting should preserve the semantic intent of the interaction: what the human accomplishes with the object. The robot may need a different posture, grasp or sequence of movements to achieve that intent. In our experiments, we use the demonstrated object trajectory, scaled for G1, as the interaction target. Object-tracking error measures how closely the robot reproduces the desired object motion, while its own motion and contact strategy remain free to change.

Retargeted references can serve as training data for downstream whole-body control policies, as studied in GMR’s analysis of retargeting and motion tracking. These policies learn to track the reference while maintaining balance and responding to physical forces. The quality of the reference therefore matters for the interactions a policy can learn.

What does a kinematic reference provide?

Carry and place a chair

Human reference

HOI-Retarget

Fig. 1. Human demonstration and HOI-Retarget reference for carrying and placing a wooden chair. Kinematic replay without physics simulation.

Push and pull a box

Human reference

HOI-Retarget

Fig. 2. Human demonstration and HOI-Retarget reference for pushing and pulling a box. Kinematic replay without physics simulation.

HOI-Retarget treats contact as part of the retargeting problem. It maps human motion and object-relative contact targets onto the humanoid, in our case Unitree G1, then optimizes the robot motion to preserve them. The result specifies how the robot and object should move. Carry and place a chair (Fig. 1) shows the intended interaction in a kinematic replay, without advancing physics.

A geometric objective still has to reconcile body pose, joint limits and contact targets on a different embodiment. These constraints can compete. In Push and pull a box (Fig. 2), the resulting body and hand configuration does not preserve a useful contact with the box. A reference can therefore require revision even before we ask a controller to execute it.

From a reference to physical execution

A kinematic reference describes the intended motion, but executing it requires the robot to balance, establish contact and support the object's weight. SPIDER addresses this gap through sampling-based optimization in physics simulation. We combine it with HOI-Retarget to evaluate whether the retargeted motions can be executed by G1 with Dex3 hands.

Physics-based optimization can improve execution, but complex interactions may require a different grasp, posture or sequence of movements. In the clothes-stand example below, HOI-Retarget + SPIDER loses balance and tips the stand. HuMoTAR refines the motion and controller to lift, transport and place the same object.

Human reference

HOI-Retarget + SPIDER

HuMoTAR

Fig. 3. Human demonstration and simulated clothes-stand manipulation. HOI-Retarget + SPIDER loses balance and tips the stand. HuMoTAR lifts, carries and places it on the floor.

We posit that accurate and generalizable retargeting requires foundational priors that capture both the semantic intent and physical requirements of a motion. The robot should accomplish what the human does to the object, while adapting its own posture, contact strategy and timing. In our experiments, the demonstration supplies an object trajectory scaled for G1. That target stays fixed; the robot’s motion and controller can change.

HuMoTAR uses an agentic harness to make these revisions through simulation feedback. An off-the-shelf model writes motion and control code, tests the interaction, and revises its solution from the observed execution. We study this process across HUMOTO and OMOMO, then examine when reports from related skills help a new interaction.

2. A study of agentic retargeting

We study three ways to use the same off-the-shelf model, GPT-6 Astra, for retargeting. Each receives the human-motion reference, the task scene and the robot interfaces. The distinction is how it proposes a solution and uses feedback.

Astra one-shot produces a motion and controller in one response. Codex works in a persistent coding session, executing and revising its solution. The Orchestrator-agent ensemble, our harness, runs four independent Codex agents within a 30-minute refinement budget and retains the valid final execution with the lowest mean ADD. Two agents also receive optional retargeting tools for contact, geometry and tracking analysis. All four explore their own solutions without exchanging intermediate results.

Executions combine the generated manipulation controller with SONIC for whole-body tracking in MuJoCo.

Fig. 4. Agent configurations evaluated in this study. Astra one-shot generates a motion and controller in one response. Codex iteratively revises its solution. HuMoTAR runs four independent agents, two with optional retargeting tools, and selects the final execution with the lowest mean ADD.

Agents can write their own analysis and control code. Two ensemble agents also receive a supplied retargeting-tools file, which they may use, modify or replace. These helpers expose the following measurements and interfaces; they do not prescribe a grasp or a refinement sequence.

Table 1. Interfaces available during refinement.

Tool or interface What it provides
Contact inspection Contact locations, normals and forces from a saved execution, with close-up views.
Geometry checks Contact gaps, surface alignment, penetration and joint-limit margins for a proposed pose.
Motion diagnostics Link-position and orientation errors, and reference, command and execution joint traces.
Object-path evaluation Mean ADD, position and orientation errors against the fixed target trajectory.
Whole-body control A frozen tracking policy driven by a motion reference.
Custom joint control Targets, gains and feedforward torques for the agent’s manipulation controller.

3. Inside the refinement process

3.1 How does the agent refine an interaction in simulation?

We begin with a human demonstration and an initial G1 motion from HOI-Retarget. The scaled object trajectory defines the interaction goal, while the robot's motion and controller remain open to refinement. Figure 5 shows how Astra turns this starting reference into a program that can be executed, inspected and revised in simulation.

Astra writes executable Python code. The code can construct wrist trajectories, solve inverse kinematics for arm and waist joints, adjust finger targets, and change the duration of individual motion phases. Across the recorded runs, agents also implement joint-impedance control, object-pose feedback and contact-dependent transitions; some try constrained inverse dynamics to compute joint torques. These strategies are implemented in the generated program, together with any revised reference trajectories and initialization.

Physical execution requires whole-body control to support the robot and hand control to manipulate the object. We use SONIC as a frozen whole-body policy, tracking a motion reference at 50 Hz. Its 29 body joints include the arms and wrists, but not the Dex3 fingers. To control the interaction, Astra writes a custom hand and arm controller in Python. This controller commands the fingers and, when needed, replaces SONIC’s arm commands with targets, gains and feedforward torques suited to the task.

The agent can also revise the motion reference supplied to SONIC so that whole-body control follows the intended arm motion. During each rollout, SONIC and the custom hand and arm controller produce the joint commands. Astra inspects the resulting execution and revises the code or reference before the next trial.

MuJoCo executes the combined joint commands and records the resulting robot and object motion. Astra can inspect rendered views, tracking errors, contact locations and forces, joint limits and collision clearances. It uses this feedback to revise the next candidate, then tests the complete interaction again. The retained result includes the executed reference motion and the controller needed to replay it. The examples in Figure 5 show the kinds of changes made within this loop.

Human demonstration and kinematic reference enter an LLM refinement loop. Generated controller code and a motion reference drive physics simulation through a whole-body policy and custom manipulation control; simulation feedback returns to the agent.

Refining contact, support and motion

(a) Starting pose
Clothes stand · before execution: Feet fitted to the floor.

Prepare the clothes-stand lift

The agent corrects foot support and hand clearance before stepping. Root height, leg posture and finger placement change while the object’s initial pose stays fixed.

(b) Contact acquisition
Right foot contacts the bin near its rim.

Tilt a bin with the right foot

Foot placement and leg motion establish contact near the rim to tilt the bin. The other foot provides support while the right foot directs the interaction.

(c) Maintaining contact
Fingers and palm support the box during lifting.

Lift and turn a large box

Both hands support the raised box through object-relative arm targets. The controller maintains the grip as the load transfers from the floor to the arms.

(d) Whole-body coordination
Upright G1 steps while carrying a wooden chair by its backrest.

Carry and place a wooden chair

Pelvis-position feedback steers the whole-body reference while arm IK follows the backrest rail. The body advances with the chair while the hand maintains its grasp.

(e) Timing and feedback
Bimanual contact during the slower rocking phase.

Rock a wooden chair

The agent slows the rocking phase and uses object-pose feedback to follow the tilt. Hand opening is staggered as the chair returns upright.

(f) Placement and release
Floor lamp · after release: Open hand after placement.

Set down a floor lamp

The agent adjusts the base orientation and waits for the lamp to settle before opening the fingers. The hand then withdraws while the floor supports the lamp.

Figure 5. HuMoTAR refinement pipeline and six examples of contact and control adjustments. Astra generates motion and controller code, evaluates it in simulation and revises it using execution feedback. Panels (a–f) show starting-pose correction, foot contact, grasp retention, coordinated transport, phase timing and object release.

3.2 Tracking error during refinement

Three side-by-side step plots of mean ADD, position error and orientation error across all 40 benchmark tasks. Dark curves show the overall mean; light curves show each dataset. The view spans 30 minutes, including no-experience continuations for the staged searches.
Fig. 6. Mean tracking errors during 30 minutes of iterative refinement on 20 HUMOTO and 20 OMOMO tasks. The dark curve averages all 40 tasks; the lighter curves show each dataset.

A kinematic reference specifies the intended motion, but execution can produce large errors when contact fails or the robot cannot support the load. Figure 6 follows the performance of the iterative refinement process over a 30-minute budget. Each step reflects a new best execution as the agent revises the motion or controller. The dark curve averages across all tasks; the lighter curves show HUMOTO and OMOMO separately. The larger drops indicate substantial improvements, followed by smaller corrections as refinement progresses.

Mean ADD selects the best execution at each point; position and orientation report that same execution. The curves begin when all tasks have a scored execution; the unrefined kinematic replay was not recorded consistently enough to mark a shared starting value.

3.3 Agentic refinement walkthrough

Here we follow the agent selected by our orchestrator harness through a complete refinement of the clothes-stand interaction from Figure 5. In this OMOMO task, the robot must lift the stand, carry it through a turn and place it on the floor. Five saved revisions trace how the agent resolves a slipping grasp, brings the body into the motion and completes release.

Human source motion
Human reference · SMPL-X
G1 places and releases the clothes stand
Orchestrator harness · final dynamics reference

Selected refinement stage
Agent’s log

Fig. 7. Five revisions of the OMOMO clothes-stand task by HuMoTAR’s selected agent. Videos show the simulated motions; agent excerpts describe the changes between trials. The highlighted curve point identifies the displayed revision and its mean ADD.

How does the reference change?

We overlay the initial kinematic reference in purple and the refined reference in blue with the physical execution in solid gray. Their spatial separation shows where the prescribed and executed robot poses differ at corresponding motion phases. The refined hand path follows the executed grasp more closely; body posture can still differ as the controller maintains balance.

Pickup, transport and placement: purple initial reference and blue refined reference overlaid on the solid gray simulated G1
(a) Reference and execution. Both rows use the same physical rollout. Insets show hand alignment with the stand.
Purple initial and blue refined finger-pad paths around the clothes stand
(b) Finger motion around the object. The video compares the two references at 0.5× speed along their shared target object path. Fading trails show recent finger-pad motion relative to the object, not measured contact forces.

Fig. 7 (continued). Initial and refined references for the clothes-stand interaction. The figure includes physical execution; the video shows reference geometry without online controller corrections.

3.4 How much refinement can a shorter budget retain?

Mean tracking errors over 30 minutes, with dotted lines marking average times to 90, 95 and 99 percent of recorded mean ADD reduction.
Fig. 8. Tracking errors and refinement milestones across 20 HUMOTO and 20 OMOMO tasks. Vertical dotted lines mark the average time to achieve 90%, 95% and 99% of each task’s 30-minute ADD reduction: 20.4, 22.3 and 23.3 minutes, respectively. Milestones are computed from mean ADD and repeated across the three metric panels.

Early stopping limits the time spent on iterative refinement while retaining the best reference motion output obtained so far. Here, the stopping condition is elapsed refinement time: at a chosen cutoff, we retain the reference motion output and its controller from the refinement stage with the lowest measured mean ADD for each task. We evaluate this condition retrospectively using the recorded executions and token usage from the full 30-minute runs.

Figure 8 marks the average time each task takes to reach 90%, 95% and 99% of its recorded ADD reduction. The 95% milestone averages 22.3 minutes, which motivates a 22.3-minute early-stopping cutoff. Stopping every task at that time retains 88.4% of the improvement on average, because individual tasks reach their milestones at different times.

4. Dynamic retargeting results

Experimental setup

We compare HOI-Retarget + SPIDER, Astra one-shot, Codex and the Orchestrator-agent ensemble on 20 HUMOTO tasks and 20 OMOMO tasks. Each dataset uses the same task list across all four methods. We report the saved final execution from each run and average equally across tasks.

Codex refines the motion with one agent. The ensemble runs four independent agents, two with optional access to the provided retargeting tools, and selects the final reference by mean ADD. Refinement is evaluated over a 30-minute budget. HOI-Retarget + SPIDER uses the frozen humanoid-object optimization configuration with G1 and Dex3. SPIDER runs at its native 60 Hz integration rate; the agent-generated controllers use the harness’s 500 Hz simulation. Both are scored against the same object targets.

Object tracking metrics

We compare the object’s pose in the dynamic humanoid–object interaction rollout with the human-reference object trajectory scaled for G1. The reference trajectory stays fixed during refinement, and the object’s size is unchanged. We compare poses along the two object paths in the order they occur, allowing the robot to move at a different speed or pause. Both paths are sampled by progress through translation and rotation, rather than elapsed time.

  • Mean ADD (cm): at each aligned pair of poses, we place the same set of object points at the simulated pose and at the reference pose, then measure the distance between each pair of corresponding points. Averaging these distances over the points and the trajectory gives mean ADD, capturing both position and orientation error in one distance.
  • Position error (cm): average Euclidean distance between the simulated and reference object origins.
  • Orientation error (°): average angle of the relative rotation between the simulated and reference object orientations.

All three metrics use the same trajectory alignment. We compute each task’s score first, then average equally across tasks; lower is better.

Results

Fig. 9. Human demonstrations and simulated humanoid–object interactions from HUMOTO and OMOMO, with executions from HuMoTAR and the available baselines.
Table 2. Object-tracking errors in the final simulated executions of 20 HUMOTO and 20 OMOMO tasks. ↓ Lower is better
MethodTasksMean ADD (cm)Position error (cm)Orientation error (°)
HUMOTO · 20 tasks
HOI-Retarget + SPIDER2056.6655.4951.22
Astra one-shot2056.0955.0362.88
Codex2020.9619.1929.32
Orchestrator-agent ensemble (ours)2014.3013.8722.07
OMOMO · 20 tasks
HOI-Retarget + SPIDER2048.3143.3652.07
Astra one-shot2069.8765.6158.35
Codex2026.5123.7028.32
Orchestrator-agent ensemble (ours)2019.6117.0627.07

5. When does experience transfer?

Comparing refinement with and without experience

We investigate whether experience from related interactions helps a robot retarget a new task. Here, G1 must raise a shelf from lying on its side to standing upright. The experience consists of reports from four other furniture tasks, including carrying a table and lifting a chair. These reports summarize the strategies tested and their outcomes.

We first run four agents for 15 minutes, two with optional access to the provided retargeting tools, and select the best valid shelf motion. From this checkpoint, we run two separate 15-minute continuations: one receives the reports, and the other continues without them. Both use the same model, tools and starting solution, with 30 minutes of total refinement. This comparison measures the benefit of the supplied experience under an equal refinement budget.

6. Discussion

Limits of the metrics

We posit that retargeting should be evaluated by the successful transfer of an interaction’s semantic intent. In this study, object tracking provides a measurable objective for preserving the intended object motion. It is possible, however, for executions with similar tracking errors to differ in how fully they accomplish the task. User studies could complement these measurements by assessing whether the intended interaction is preserved. LLM-based judges could also supply semantic feedback during refinement and evaluation. We encourage the community to explore these directions and assess their agreement with human judgments.

RL fine-tuning

Humanoid references can serve as training data for downstream whole-body policies. Our results demonstrate the promise of agentic refinement using off-the-shelf models and low-level controllers, without training a new policy for each interaction. Reinforcement-learning fine-tuning could build on these references to improve robustness to disturbances and variation in the task scene. Downstream policy training and deployment remain to be evaluated. We hope this work encourages the community to investigate agentic methods as a source of reference motions for learning interactive humanoid systems.

Failure cases in dynamic retargeting

Our approach still struggles with thin objects and difficult contact geometries. In the plate example, the robot lifts the plate but does not complete the human’s swirling motion. Across the recorded attempts, agents revise finger placement and try two-handed support, yet the grasp remains unreliable during rotation. We observe that contact strategies must account for the robot’s hand morphology. The Dex3 fingers must establish opposing contacts around a thin, sloped rim, which appears to contribute to the difficulty of this task. Motion and controller refinement do not yet resolve this limitation.

Human reference

HuMoTAR

Fig. 11. Human demonstration and HuMoTAR execution of the plate-swirling task. The robot partially lifts the plate but does not complete the demonstrated swirling motion.

References

  1. HOI-Retarget. Jihwan Shin et al. HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction. 2026. Paper · Project.
  2. GMR. Joao Pedro Araujo et al. Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking. 2025. Paper · Code.
  3. SPIDER. Chaoyi Pan et al. SPIDER: Scalable Physics-Informed Dexterous Retargeting. 2025; revised 2026. Paper · Code.
  4. SONIC. Zhengyi Luo et al. SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control. 2025 preprint; Science Robotics, 2026. Paper · Project.
  5. HUMOTO. Jiaxin Lu et al. HUMOTO: A 4D Dataset of Mocap Human Object Interactions. 2025. Paper · Project.
  6. OMOMO. Jiaman Li, Jiajun Wu and C. Karen Liu. Object Motion Guided Human Motion Synthesis. SIGGRAPH Asia, 2023. Paper · Project and dataset.
  7. MuJoCo. Emanuel Todorov, Tom Erez and Yuval Tassa. MuJoCo: A physics engine for model-based control. IROS, 2012. Paper · Current simulator.
  8. SMPL-X. Georgios Pavlakos et al. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. CVPR, 2019. Paper and model.
  9. GPT-6 Astra. OpenAI. GPT-6 Astra: A new generation of intelligence. Model announcement.
  10. Codex. OpenAI. Codex CLI. Official coding-agent repository. This citation refers to the agent software; GPT-6 Astra is the model used in our experiments.
  11. G1 and Dex3. Unitree Robotics. G1 platform · Official robot simulation models.

Appendix

How does HuMoTAR compare on kinematic retargeting?

We compare GMR, HOI-Retarget and the Orchestrator-agent ensemble on the same task set.

We follow HOI-Retarget’s evaluation protocol (Appendix A): body pose deviation, link velocity direction error, contact-point gap and relative hand-orientation change measure how well the robot preserves the human motion and object-relative contacts. We also report body jerk as a smoothness measure, the mean magnitude of the third time derivative of the palm, ankle, head and pelvis positions, estimated by finite differences. Lower values indicate smoother motion.

Fig. A1. Hand–object contact in five kinematic retargeting tasks. GMR, HOI-Retarget and HuMoTAR are shown from matched viewpoints and motion intervals. Reported contact gaps are averaged over each complete motion. Lower is better.
Table A1. Kinematic reference errors on 20 HUMOTO and 20 OMOMO tasks using G1 with Dex3 hands. ↓ Lower is better
MethodTasksBody pose deviation (°)Velocity direction error (°)Contact-point gap (mm)Hand orientation change (°)Body jerk (m/s³)
HUMOTO · 20 tasks
GMR2014.6820.4164.727.3883.40
HOI-Retarget2015.1828.0836.587.4125.93
Orchestrator-agent ensemble (ours)208.9810.654.270.239.18
OMOMO · 20 tasks
GMR2017.4221.53136.631.5077.63
HOI-Retarget2018.9426.1031.314.9334.68
Orchestrator-agent ensemble (ours)208.5611.073.530.209.99

HUMOTO: contact gap uses 16 tasks; hand orientation uses 16. Each metric uses the same applicable tasks across all three methods.

Can a text description replace a motion reference?

We give a single Codex agent a detailed task description, the scene and an initial robot–object state, without a human motion reference or target object trajectory. The agent has 30 minutes to refine an execution that follows the description, using SONIC for the legs and waist and custom arm and hand control. These five examples show the supplied commands and the resulting simulated motions.

Lift and inspect a basket

Text command

A basket rests on the floor in front of the person. Bend toward it and reach down with the right hand to take its handle. Lift the basket clear of the floor while straightening the body. Bring it in front of the torso, then use the other hand to help steady and turn it for inspection. Tilt and rotate the basket so its opening and different sides can be examined, rather than simply holding it motionless. Keep it under control throughout the lift and the changes in orientation. Finish standing with the basket held in front of the body; the demonstrated sequence ends while it is still being held.
Simulated output for Lift and inspect a basket
Simulated execution · normal speed

Cite this blog

Narayanan Palghat Parameswaran, Jay Karhade, Haoyang He, Neehar Peri, Jay Patrikar, Dong-Ki Kim, Shayegan Omidshafiei, Sebastian Scherer, Deva Ramanan. HuMoTAR: Semantic Humanoid Retargeting with Foundational Priors. 2026.

Download BibTeX

@misc{parameswaran2026humotar,
  title = {HuMoTAR: Semantic Humanoid Retargeting with Foundational Priors},
  author = {Palghat Parameswaran, Narayanan and Karhade, Jay and He, Haoyang
            and Peri, Neehar and Patrikar, Jay and Kim, Dong-Ki
            and Omidshafiei, Shayegan and Scherer, Sebastian and Ramanan, Deva},
  year = {2026},
  howpublished = {Project blog},
  url = {https://narayananpp.github.io/semantic-humanoid-retargeting-blog/}
}