Real-to-Sim 6-DoF Object Pose Estimation

Course project for 3D Vision at ETH Zurich (Spring 2026), with Jihwan Shin and Hugo De la Riva Fernandez.

Pipeline overview: kinematic replay reconstructs the demonstration from RGB video and robot proprioception, then a dynamic replay in simulation recovers physically consistent object trajectories with rich information (contact forces, segmentation, multi-view renders).

The project estimates 6-degree-of-freedom object pose from a single egocentric RGB video stream — no depth sensor required. The system tracks the object’s pose through a complete manipulation sequence (grasping, carrying, dropping) and generates simulation-ready trajectories for replay in Isaac Sim.

The pipeline combines:

Object segmentation paired with monocular depth estimation
A refined pose anchor established at frame 0
In-hand tracking using gripper kinematics constraints
Post-release 6D refinement with outlier rejection and smoothing
Trajectory composition into per-frame 4×4 transformation matrices

My contribution: I implemented the dynamics-consistent trajectory replay and the residual RL pipeline for recovering a dynamically feasible rollout. Since neither the estimated object trajectory nor the recorded robot trajectory is physically valid in isolation (occlusion, depth ambiguity, sensor latency), a residual policy emits bounded corrections on the nominal robot configuration — applied as PD joint-position targets — to jointly refine the robot trajectory explicitly and the object trajectory implicitly through simulated contact dynamics in Isaac Sim, inspired by ManipTrans (Li et al., CVPR 2025).

RL-refined simulation rollout (top) reproducing the recorded real-world manipulation sequence (bottom).

Residual policy training across 64 GPU-parallelized Isaac Sim environments.

Project page: hudela390.github.io/real2sim-6dpose

Code: RGBTrack-3DV- (pose estimation) · ManipTrans_isaacsim (residual RL refinement)