Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization

Rendering-based tracking is attractive because it directly compares a hypothesized robot configuration with visual observations, but iterative rendering can be slow and sensitive to local minima. We use the covariance matrix adaptation evolution strategy (CMA-ES) together with GPU-batched rendering to evaluate pose candidates in parallel. The resulting framework jointly estimates the six-degree-of-freedom instrument pose and visible joint angles. Its direct render-and-match formulation is also simple to adapt when joint readings are unavailable or two instruments must be tracked together.

Across synthetic and real-world experiments, the method improves both reconstruction accuracy and runtime over differentiable-rendering baselines. In the online comparison reported in the paper, it reaches 43.34 FPS while reducing mask error relative to a particle-filter baseline.

Read the paper on arXiv or see the brief project note.

Recommended citation: Hanyang Hu, Zekai Liang, Florian Richter, and Michael C. Yip. (2026). "Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
Download Paper