Direct Camera-Only Bundle Adjustment for 3-D Textured Colon Surface Reconstruction Based on Pre-Operative Model Shuai Zhang, Liang Zhao, Shoudong Huang, Evangelos B.Mazomenos, Danail Stoyanov IEEE Transactions on Medical Robotics and Bionics, 2025 This paper addresses the problem of reconstructing textured colon surface maps using a sequence of monocular colonoscopic images together with a 3D colon mesh model that has been segmented in CT colonography. The problem is formulated as a direct bundle adjustment (BA) problem which simultaneously optimizes all camera poses and the intensity of vertices on the pre-operative mesh model. This optimization is achieved by maximizing photometric consistency among multiple views of 2D images and the pre-operative 3D mesh model. The key properties of our proposed direct BA formulation involve eliminating the need for reference image specification, data association (feature extraction and matching), and image depth information. Thus, the proposed method is particularly suitable for scenarios where distinct features and image depth are not available, such as 2D colonoscopic images. Furthermore, we have proven that solving the proposed direct BA using the Gauss-Newton (GN) algorithm has the merit of optimizing camera poses only, which is equivalent to optimizing camera poses and the intensities of 3D vertices on the mesh together. Thus, a direct camera-only BA algorithm is proposed and used for 3D textured colon reconstruction from textureless 2D colonoscopic images. Validations using simulation, phantom, and in-vivo datasets are performed to demonstrate the accuracy and feasibility of the proposed algorithm.
StereoMamba: Real-Time and Robust Intraoperative Stereo Disparity Estimation via Long-Range Spatial Dependencies Xu Wang, Jialang Xu, Shuai Zhang, Baoru Huang, Danail Stoyanov, Evangelos B. Mazomenos IEEE Robotics and Automation Letters, 2025 Stereo disparity estimation is crucial for obtaining depth information in robot-assisted minimally invasive surgery (RAMIS). While current deep learning methods have made significant advancements, challenges remain in achieving an optimal balance between accuracy, robustness, and inference speed. To address these challenges, we propose the StereoMamba architecture, which is specifically designed for stereo disparity estimation in RAMIS. Our approach is based on a novel Feature Extraction Mamba (FE-Mamba) module, which enhances long-range spatial dependencies both within and across stereo images. To effectively integrate multi-scale features from FE-Mamba, we then introduce a novel Multidimensional Feature Fusion (MFF) module. Experiments against the state-of-the-art on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ex-vivo</i> SCARED benchmark demonstrate that StereoMamba achieves superior performance on EPE of 2.64 px and depth MAE of 2.55 mm, the second-best performance on Bad2 of 41.49% and Bad3 of 26.99%, while maintaining an inference speed of 21.28 FPS for a pair of high-resolution images (1280×1024), striking the optimum balance between accuracy, robustness, and efficiency. Furthermore, by comparing synthesized right images, generated from warping left images using the generated disparity maps, with the actual right image, StereoMamba achieves the best average SSIM (0.8970) and PSNR (16.0761), exhibiting strong zero-shot generalization on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-vivo</i> RIS2017 and StereoMIS datasets.
Adjunct Tools for Colonoscopy Enhancement: A Comprehensive Review Neri Niccolò Dei, Evangelos B. Mazomenos, Shuai Zhang, Sophia Bano, Josè M. M. Montiel, Danail Stoyanov, Gastone Ciuti IEEE Transactions on Medical Robotics and Bionics, 2025 Colonoscopy is considered the gold standard for detecting and diagnosing colorectal cancer (CRC), which is the second most common cause of cancer-related deaths worldwide. While colonoscopy is generally safe and effective at reducing CRC mortality, lesions can be missed during procedures, with adverse impacts on the patient. Latest innovations in hardware and software led to the development of adjunct tools for complementing standard colonoscopy to ensure optimal outcomes. Such tools aim to enhance the detection of lesions, standardize procedures, enhance safety, and minimize discomfort. Ultimately, they contribute to reducing the morbidity and mortality rates associated with CRC. This survey comprehensively explores both clinically tested and emerging advanced hardware and software adjunct tools, categorizing them based on their role in targeting three clinical challenges: mucosal visualization, lesion detection and classification, and navigation and procedure assessment. Moreover, this analysis allows exploring synergistic strategies for the future of the practice, with a focus on the promising role of AI-embedded robotic technologies.
Adaptive Dual-Axis Style-Based Recalibration Network With Class-Wise Statistics Loss for Imbalanced Medical Image Classification Xiaoqing Zhang, Zunjie Xiao, Jingzhe Ma, Xiao Wu, Jilu Zhao, Shuai Zhang, Runzhi Li, Yi Pan, Jiang Liu IEEE Transactions on Image Processing, 2025 Salient and small lesions (e.g., microaneurysms on fundus) both play significant roles in real-world disease diagnosis under medical image examinations. Although deep neural networks (DNNs) have achieved promising medical image classification performance, they often have limitations in capturing both salient and small lesion information, restricting performance improvement in imbalanced medical image classification. Recently, with the advent of DNN-based style transfer in medical image generation, the roles of clinical styles have attracted great interest, as they are crucial indicators of lesions. Motivated by this observation, we propose a novel Adaptive Dual-Axis Style-based Recalibration (ADSR) module, leveraging the potential of clinical styles to guide DNNs in effectively learning salient and small lesion information from a dual-axis perspective. ADSR first emphasizes salient lesion information via global style-based adaptation, then captures small lesion information with pixel-wise style-based fusion. We construct an ADSR-Net for imbalanced medical image classification by stacking multiple ADSR modules. Additionally, DNNs typically adopt cross-entropy loss for parameter optimization, which ignores the impacts of class-wise predicted probability distributions. To address this, we introduce a new Class-wise Statistics Loss (CWS) combined with CE to further boost imbalanced medical image classification results. Extensive experiments on five imbalanced medical image datasets demonstrate not only the superiority of ADSR-Net and CWS over state-of-the-art (SOTA) methods but also their improved confidence calibration results. For example, ADSR-Net with the proposed loss significantly outperforms CABNet50 by 21.39% and 27.82% in F1 and B-ACC while reducing 3.31% and 4.57% in ECE and BS on ISIC2018.
SLAM-TKA: Simultaneously Localizing X-Ray Device and Mapping Pins in Conventional Total Knee Arthroplasty Shuai Zhang, Liang Zhao, Shoudong Huang, Hua Wang, Qi Luo, Qi Hao, Danail Stoyanov IEEE Transactions on Medical Robotics and Bionics, 2024 This paper presents a novel simultaneous localization and mapping (SLAM) technique, termed SLAM-TKA, for assisting total knee arthroplasty (TKA), a highly effective orthopaedic surgery that replaces arthritic or dysfunctional joint surfaces with knee prostheses. Our proposed SLAM algorithm uses information from a pre-operative tibia CT scan, intra-operative 2D X-ray images, and a trocar pin 3D mesh model to simultaneously localise the X-ray device and map the two trocar pins. Then, the estimated pins are used to evaluate the accuracy of the bone resection plane before the actual bone cutting, which plays a crucial role in precisely implanting the knee prostheses. To ensure high accuracy and robustness of the proposed SLAM algorithm, three energy terms are proposed and used together to align the edge observations of the tibia, fibula and pins on the intra-operative X-ray images and their corresponding pre-operative 3D mesh models in both 2D and 3D space. To enable the proposed iteration-based SLAM algorithm to be implemented in real-time such that the evaluation processing does not interrupt much on the workflow of TKA, the data association of edge correspondences matching and exhausted points-to-mesh distance calculation are pre-computed using the signed distance field method. Simulations are used to evaluate the accuracy and robustness of the proposed algorithm, and the experiments using in-vivo datasets from five patients demonstrate the high accuracy and efficiency in practice. The code and datasets are released at <uri>https://github.com/zsustc/SLAM-TKA</uri>.
Is it feasible to develop a supervised learning algorithm incorporating spinopelvic mobility to predict impingement in patients undergoing total hip arthroplasty? A proof-of-concept study Andreas Fontalis, Baixiang Zhao, Pierre Putzeys, Fabio Mancino, Shuai Zhang, Thomas Vanspauwen, Fabrice Glod, Ricci Plastow, Evangelos Mazomenos, Fares S. Haddad Bone and Joint Open, 2024 AimsPrecise implant positioning, tailored to individual spinopelvic biomechanics and phenotype, is paramount for stability in total hip arthroplasty (THA). Despite a few studies on instability prediction, there is a notable gap in research utilizing artificial intelligence (AI). The objective of our pilot study was to evaluate the feasibility of developing an AI algorithm tailored to individual spinopelvic mechanics and patient phenotype for predicting impingement.MethodsThis international, multicentre prospective cohort study across two centres encompassed 157 adults undergoing primary robotic arm-assisted THA. Impingement during specific flexion and extension stances was identified using the virtual range of motion (ROM) tool of the robotic software. The primary AI model, the Light Gradient-Boosting Machine (LGBM), used tabular data to predict impingement presence, direction (flexion or extension), and type. A secondary model integrating tabular data with plain anteroposterior pelvis radiographs was evaluated to assess for any potential enhancement in prediction accuracy.ResultsWe identified nine predictors from an analysis of baseline spinopelvic characteristics and surgical planning parameters. Using fivefold cross-validation, the LGBM achieved 70.2% impingement prediction accuracy. With impingement data, the LGBM estimated direction with 85% accuracy, while the support vector machine (SVM) determined impingement type with 72.9% accuracy. After integrating imaging data with a multilayer perceptron (tabular) and a convolutional neural network (radiograph), the LGBM’s prediction was 68.1%. Both combined and LGBM-only had similar impingement direction prediction rates (around 84.5%).ConclusionThis study is a pioneering effort in leveraging AI for impingement prediction in THA, utilizing a comprehensive, real-world clinical dataset. Our machine-learning algorithm demonstrated promising accuracy in predicting impingement, its type, and direction. While the addition of imaging data to our deep-learning algorithm did not boost accuracy, the potential for refined annotations, such as landmark markings, offers avenues for future enhancement. Prior to clinical integration, external validation and larger-scale testing of this algorithm are essential.Cite this article: Bone Jt Open 2024;5(8):671–680.
3D Reconstruction of Tibia and Fibula using One General Model and Two X-ray Images Kai Pan, Shuai Zhang, Liang Zhao, Shoudong Huang, Yanhao Zhang, Hua Wang, Qi Luo Proceedings IEEE International Conference on Robotics and Automation, 2023 The 3D reconstruction of patient specific bone models plays a crucial role in orthopaedic surgery for clinical evaluation, surgical planning and precise implant design or selection. This paper considers the problem of reconstructing a patient-specific 3D tibia and fibula model from only two 2D X-ray images and one 3D general model segmented from the lower leg CT scans of one randomly selected patient. Currently, the bone 3D reconstruction mainly relies on computed tomography (CT) and magnetic resonance imaging (MRI) scanning-based mode segmentation which result in high radiation exposure or expensive costs. While, the proposed algorithm can accurately and efficiently deform a 3D general model to achieve a patient-specific 3D model that matches the patient's tibia and fibula projections in two 2D X-rays. The algorithm undergoes a preliminary deformation, 2D contour registration, and opti-misation based on the deformation graph that represents the shape deformation of models. Evaluations using simulations, cadaver and in-vivo experiments demonstrate that the proposed algorithm can effectively reconstruct the patient's 3D tibia and fibula surface model with high accuracy.
A Template-Based 3D Reconstruction of Colon Structures and Textures from Stereo Colonoscopic Images Shuai Zhang, Liang Zhao, Shoudong Huang, Menglong Ye, Qi Hao IEEE Transactions on Medical Robotics and Bionics, 2021 This article presents a framework for 3D reconstruction of colonic surface using stereo colonoscopic images. Due to the limited overlaps between consecutive frames and the nonexistence of large loop closures during a normal screening colonoscopy, the state-of-art simultaneous localization and mapping (SLAM) algorithms cannot be directly applied to this scenario, thus a colon model segmented from CT scans is used together with the colonosocopic images to achieve the colon 3D reconstruction with high accuracy. The proposed framework includes 3D scan (point cloud with RGB information) reconstruction from stereo images, a visual odometry (VO) based camera pose initialization module, a 3D registration scheme for matching texture scans to the segmented colon model, and a barycentric-based texture rendering module for mapping textures from colonoscopic images to the reconstructed colonic surface. A realistic simulator is developed using Unity to simulate the procedures of colonoscopy and used to provide experimental datasets in different scenarios. Experimental results demonstrate the good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness. Currently, the framework requires a pre-operative colon model as the template for colon reconstruction and can reconstruct 3D colon maps when the colon has no large deformation and the colon structure is clearly visible. The datasets used in this article and the developed simulator are made publicly available for other researchers to use (https://github.com/zsustc/colon_reconstruction_dataset).
3D Acetabular Surface Reconstruction from 2D Pre-operative X-Ray Images Using SRVF Elastic Registration and Deformation Graph (MICCAI, Oral) S Zhang, J Wang, X Wang, S Konan, D Stoyanov, EB Mazomenos International Conference on Medical Image Computing and Computer-Assisted … , 2025 2025
Adjunct tools for colonoscopy enhancement: a comprehensive review NN Dei, EB Mazomenos, S Zhang, S Bano, JMM Montiel, D Stoyanov, ... IEEE Transactions on Medical Robotics and Bionics , 2025 2025 Citations: 2
Endolrmgs: Complete endoscopic scene reconstruction combining large reconstruction modelling and gaussian splatting X Wang, S Zhang, B Huang, D Stoyanov, EB Mazomenos arXiv preprint arXiv:2503.22437 , 2025 2025 Citations: 3
Adaptive dual-axis style-based recalibration network with class-wise statistics loss for imbalanced medical image classification X Zhang, Z Xiao, J Ma, X Wu, J Zhao, S Zhang, R Li, Y Pan, J Liu IEEE Transactions on Image Processing , 2025 2025 Citations: 30
StereoMamba: Real-Time and Robust Intraoperative Stereo Disparity Estimation via Long-Range Spatial Dependencies. X Wang, J Xu, S Zhang, B Huang, D Stoyanov, EB Mazomenos IEEE Robotics and Automation Letters, 10682-10689 , 2025 2025
Direct Camera-Only Bundle Adjustment for 3-D Textured Colon Surface Reconstruction Based on Pre-Operative Model S Zhang, L Zhao, S Huang, EB Mazomenos, D Stoyanov IEEE Transactions on Medical Robotics and Bionics 7 (1), 242-253 , 2024 2024 Citations: 1
Gaussian pancakes: geometrically-regularized 3d gaussian splatting for realistic endoscopic reconstruction S Bonilla, S Zhang, D Psychogyios, D Stoyanov, F Vasconcelos, S Bano International Conference on Medical Image Computing and Computer-Assisted … , 2024 2024 Citations: 38
SLAM-TKA: Simultaneously Localizing X-Ray Device and Mapping Pins in Conventional Total Knee Arthroplasty S Zhang, L Zhao, S Huang, H Wang, Q Luo, Q Hao, D Stoyanov IEEE Transactions on Medical Robotics and Bionics 6 (4), 1526-1541 , 2024 2024 Citations: 2
Is it feasible to develop a supervised learning algorithm incorporating spinopelvic mobility to predict impingement in patients undergoing total hip arthroplasty?: a proof-of … A Fontalis, B Zhao, P Putzeys, F Mancino, S Zhang, T Vanspauwen, ... Bone & Joint Open 5 (8), 671-680 , 2024 2024 Citations: 8
3d reconstruction of tibia and fibula using one general model and two x-ray images K Pan, S Zhang, L Zhao, S Huang, Y Zhang, H Wang, Q Luo 2023 IEEE International Conference on Robotics and Automation (ICRA), 4732-4738 , 2023 2023 Citations: 5
3D Reconstruction of Colon Structures and Textures from Colonoscopic Videos S Zhang PQDT-Global , 2023 2023
SLAM-TKA: Real-time intra-operative measurement of tibial resection plane in conventional total knee arthroplasty (MICCAI Oral & Travel award) S Zhang, L Zhao, S Huang, H Wang, Q Luo, Q Hao International Conference on Medical Image Computing and Computer-Assisted … , 2022 2022 Citations: 6
3D reconstruction of deformable colon structures based on preoperative model and deep neural network S Zhang, L Zhao, S Huang, R Ma, B Hu, Q Hao 2021 IEEE International Conference on Robotics and Automation (ICRA), 1875-1881 , 2021 2021 Citations: 12
A template-based 3D reconstruction of colon structures and textures from stereo colonoscopic images S Zhang, L Zhao, S Huang, M Ye, Q Hao IEEE Transactions on Medical Robotics and Bionics 3 (1), 85-95 , 2020 2020 Citations: 53
Linear Bayesian filter based low-cost UWB systems for indoor mobile robot localization S Zhang, R Han, W Huang, S Wang, Q Hao 2018 IEEE SENSORS, 1-4 , 2018 2018 Citations: 12
An integrated uav navigation system based on geo-registered 3d point cloud S Zhang, S Wang, C Li, G Liu, Q Hao 2017 IEEE International Conference on Multisensor Fusion and Integration for … , 2017 2017 Citations: 3
A camera-based real-time polarization sensor and its application to mobile robot navigation S Zhang, H Liang, H Zhu, D Wang, B Yu 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014 … , 2014 2014 Citations: 13
A bionic camera-based polarization navigation sensor D Wang, H Liang, H Zhu, S Zhang Sensors 14 (7), 13006-13023 , 2014 2014 Citations: 90
MOST CITED SCHOLAR PUBLICATIONS
A bionic camera-based polarization navigation sensor D Wang, H Liang, H Zhu, S Zhang Sensors 14 (7), 13006-13023 , 2014 2014 Citations: 90
A template-based 3D reconstruction of colon structures and textures from stereo colonoscopic images S Zhang, L Zhao, S Huang, M Ye, Q Hao IEEE Transactions on Medical Robotics and Bionics 3 (1), 85-95 , 2020 2020 Citations: 53
Gaussian pancakes: geometrically-regularized 3d gaussian splatting for realistic endoscopic reconstruction S Bonilla, S Zhang, D Psychogyios, D Stoyanov, F Vasconcelos, S Bano International Conference on Medical Image Computing and Computer-Assisted … , 2024 2024 Citations: 38
Adaptive dual-axis style-based recalibration network with class-wise statistics loss for imbalanced medical image classification X Zhang, Z Xiao, J Ma, X Wu, J Zhao, S Zhang, R Li, Y Pan, J Liu IEEE Transactions on Image Processing , 2025 2025 Citations: 30
A camera-based real-time polarization sensor and its application to mobile robot navigation S Zhang, H Liang, H Zhu, D Wang, B Yu 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014 … , 2014 2014 Citations: 13
3D reconstruction of deformable colon structures based on preoperative model and deep neural network S Zhang, L Zhao, S Huang, R Ma, B Hu, Q Hao 2021 IEEE International Conference on Robotics and Automation (ICRA), 1875-1881 , 2021 2021 Citations: 12
Linear Bayesian filter based low-cost UWB systems for indoor mobile robot localization S Zhang, R Han, W Huang, S Wang, Q Hao 2018 IEEE SENSORS, 1-4 , 2018 2018 Citations: 12
Is it feasible to develop a supervised learning algorithm incorporating spinopelvic mobility to predict impingement in patients undergoing total hip arthroplasty?: a proof-of … A Fontalis, B Zhao, P Putzeys, F Mancino, S Zhang, T Vanspauwen, ... Bone & Joint Open 5 (8), 671-680 , 2024 2024 Citations: 8
SLAM-TKA: Real-time intra-operative measurement of tibial resection plane in conventional total knee arthroplasty (MICCAI Oral & Travel award) S Zhang, L Zhao, S Huang, H Wang, Q Luo, Q Hao International Conference on Medical Image Computing and Computer-Assisted … , 2022 2022 Citations: 6
3d reconstruction of tibia and fibula using one general model and two x-ray images K Pan, S Zhang, L Zhao, S Huang, Y Zhang, H Wang, Q Luo 2023 IEEE International Conference on Robotics and Automation (ICRA), 4732-4738 , 2023 2023 Citations: 5
Endolrmgs: Complete endoscopic scene reconstruction combining large reconstruction modelling and gaussian splatting X Wang, S Zhang, B Huang, D Stoyanov, EB Mazomenos arXiv preprint arXiv:2503.22437 , 2025 2025 Citations: 3
An integrated uav navigation system based on geo-registered 3d point cloud S Zhang, S Wang, C Li, G Liu, Q Hao 2017 IEEE International Conference on Multisensor Fusion and Integration for … , 2017 2017 Citations: 3
Adjunct tools for colonoscopy enhancement: a comprehensive review NN Dei, EB Mazomenos, S Zhang, S Bano, JMM Montiel, D Stoyanov, ... IEEE Transactions on Medical Robotics and Bionics , 2025 2025 Citations: 2
SLAM-TKA: Simultaneously Localizing X-Ray Device and Mapping Pins in Conventional Total Knee Arthroplasty S Zhang, L Zhao, S Huang, H Wang, Q Luo, Q Hao, D Stoyanov IEEE Transactions on Medical Robotics and Bionics 6 (4), 1526-1541 , 2024 2024 Citations: 2
Direct Camera-Only Bundle Adjustment for 3-D Textured Colon Surface Reconstruction Based on Pre-Operative Model S Zhang, L Zhao, S Huang, EB Mazomenos, D Stoyanov IEEE Transactions on Medical Robotics and Bionics 7 (1), 242-253 , 2024 2024 Citations: 1
3D Acetabular Surface Reconstruction from 2D Pre-operative X-Ray Images Using SRVF Elastic Registration and Deformation Graph (MICCAI, Oral) S Zhang, J Wang, X Wang, S Konan, D Stoyanov, EB Mazomenos International Conference on Medical Image Computing and Computer-Assisted … , 2025 2025
StereoMamba: Real-Time and Robust Intraoperative Stereo Disparity Estimation via Long-Range Spatial Dependencies. X Wang, J Xu, S Zhang, B Huang, D Stoyanov, EB Mazomenos IEEE Robotics and Automation Letters, 10682-10689 , 2025 2025
3D Reconstruction of Colon Structures and Textures from Colonoscopic Videos S Zhang PQDT-Global , 2023 2023
