Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking
Chi Zhang,Wangru Zhu,Jianqing Peng,Yu Han,Wanquan Liu
DOI: https://doi.org/10.1016/j.measurement.2023.112658
IF: 5.6
2023-04-01
Measurement
Abstract:In routine laparoscopic surgeries, the fatigue operation and improper chief-assistant coordination of the assistant surgeon both reduce the stability of the endoscopic field of view (EFOV) and the surgical efficiency. In contrast, the assistant endoscope-holding robot (AEHR) can stably hold the endoscope and adjust the pose of endoscopic image sensor. However, the traditional method simplifies the tracking of surgical instrument tips (SITs) to the tracking of feature points, and lacks the depth optimization of monocular endoscopic image senor. Based on them, this paper proposes a surgical instrument tracking control method based on visual tracking features (VTFs) and hand-eye coordination (HEC) with feedback of multiple sensors, which can control the depth and axial rotational degrees of freedom (DOFs) of endoscope while tracking SITs. Firstly, the motion mapping Jacobian matrix between the endoscopic image sensor and the endoscope-holding manipulator (EHM) end-effector is constructed with joint encoder feedbacks, and the VTFs of endoscopic camera feedback are extracted based on the deep segmentation model of SITs. Secondly, the position adjustment model based on VTFs and the attitude adjustment model based on HEC are established. Thirdly, a complete set of visual servo control (VSC) software and hardware system for AEHRs is designed. Finally, the correctness, effectiveness and operability of the proposed method are comprehensively verified through the simulation and experiment of surgical instrument tracking. The visual tracking distance and radius error of endoscopic image sensor frames are reduced by no less than 34.6% and 42.59% under different experimental backgrounds, respectively.
engineering, multidisciplinary,instruments & instrumentation