Real-time Surgical Instrument Tracking with Evolutionary Optimization
Reliable pose estimation for surgical instruments is difficult for a rather geometric reason: only part of a long articulated tool is visible through the endoscopic camera, while cable-driven joints introduce kinematic offsets that are not well represented by the robot’s raw joint readings. Keypoint-based methods can fail under blur, occlusion, and limited training data. Rendering-based methods use stronger geometric information, but conventional gradient-based optimization is often slow and sensitive to initialization.
In our IROS 2026 paper, with Zekai Liang, Florian Richter, and Michael C. Yip, we replace the gradient-based inner loop with the covariance matrix adaptation evolution strategy (CMA-ES). At each iteration, CMA-ES samples a population of candidate states from a Gaussian search distribution. A GPU renderer evaluates their silhouettes in one batch, and the resulting scores update the search distribution for the next iteration. The optimized state contains the six-degree-of-freedom instrument pose together with the visible wrist and jaw angles.
The objective combines silhouette agreement with tool-tip reprojection:
\[\mathcal{L}(\Theta) = \lambda_{\mathrm{mask}}\,\mathcal{L}_{\mathrm{mask}}(\Theta) + \lambda_{\mathrm{kpt}}\,\mathcal{L}_{\mathrm{kpt}}(\Theta),\]where $\Theta$ denotes the pose and joint parameters. A constant-velocity Kalman filter provides temporal smoothing and initializes the search distribution for the next frame. CMA-ES removes the need to differentiate through the renderer. Separately, the direct render-and-match formulation is simpler than feature-based tracking: joint readings can be omitted from the state initialization, and the state can be extended to two instruments without designing a new set of intermediate features. This is what enables joint angle-free and bi-manual tracking within the same framework.
The most satisfying result is that the computationally simpler black-box formulation is also faster in practice. With three optimization iterations per frame, our method requires about 37% of the runtime of the differentiable-rendering baseline while improving pose reconstruction. In the fully online comparison, including segmentation, it reaches 43.34 FPS with a mask error of 0.1177, compared with 19.22 FPS and 0.2476 for the particle-filter baseline.
There are still clear limitations. Rendering-based estimation depends on accurate segmentation, and segmentation errors propagate directly into the recovered pose. The real-world dataset also does not contain frequent instrument-instrument occlusion, so the most difficult bi-manual cases remain insufficiently tested. Nevertheless, the project convinced me that evolutionary optimization is a practical tool for real-time inverse graphics when candidate evaluations can be parallelized efficiently.