The Fusion of Robotics and Imaging: A Vision of the Future

Image Segmentation

Segmentation is the process of assigning labels, or objects of interest, to pixels or regions in a digital image. In the medical field, this translates into identifying organs, bones, tissue, tools, and other anatomical parts of interest in a medical image for planning and diagnostics. It can be particularly useful to separate tissue from organs, analyze cardiac images, detect tumors,²¹ and even create 3D representations of airways for bronchoscopy.²²

Segmentation can be achieved semi-autonomously by a combination of manual work and classic computational techniques of image processing such as thresholding, boundary extraction, and region growing.^23,24 It usually requires specialized tools and domain expertise to tune parameters that control the quality of the result. However, recently, learning-based methods have gained popularity in efforts to achieve efficient and automatic medical image segmentation.

Notable methods include deep learning-based approaches such as convolutional neural networks (CNNs) and generative adversarial networks (GANs) that harness the increased capabilities of machine learning and existing datasets to create models for automatic image segmentation.²⁵ In particular, the U-Net architecture has achieved tremendous success in image segmentation due to its flexibility, modularity, and ability to process images from different modalities, leading to many extensions and enhancements for a wide variety of applications.²⁶ It consists of an encoder-decoder architecture made of convolutional layers with skip connections to improve localization of high-level semantic features. U-Net has been used to improve diagnostics, automatic cardiac diagnosis, radiotherapy of head and neck CT scans, and tumor detection, among many others.^27,28,29,30 TotalSegmentator is a U-Net type of network trained to robustly segment CT and MR images of more than a hundred different anatomic structures.^31,32 It has been trained on a large dataset comprising thousands of patients—using diverse equipment and protocols at different institutions—and the model is open-source for free use.³³

One limitation of deep learning-based models is that they are task-specific and cannot generalize across anatomical structures; ie, they can only be used for anatomy that was considered in the training stage and cannot generalize to unseen structures. However, inspired by the success of recent segmentation foundation models for natural images,^34,35 Ma et al. proposed MedSAM, a foundation model to enable universal medical image segmentation, which can achieve state-of-the-art performance in multiple tasks and image modalities.³⁶ This model was trained on a dataset of over 1 million image-masks pairs from 10 different imaging modalities. The hope is to create models that can be trained once and then used for a wide variety of segmentation tasks.

Image Registration

Image registration is the process of aligning coordinate systems in different medical images that may have come from different techniques, modalities, or perspectives.¹⁰ Usually, the registration is done based on relevant features in both images, defining a similarity metric and finding the parameters of the transformation that optimize such metrics. For example, Jasper et al. used landmarks to register preoperative 3D models and intraoperative US images for active liver compensation with high accuracy.³⁷ Recently, Xihan et al. tested the feasibility of registering preoperative CT with US for spine surgery using a novel point cloud-based representation.²⁰

Deep learning-based registration methods have also been developed to tackle a diverse range of tasks such as 2D to 3D MRI, CT to cone beam CT, MRI to US, and CT to MRI, among others, where the most common network architectures include CNNs, GANs, U-Net, and encoder-decoders.^{38,39,40,41,42,43,44,45,46} A variety of organs are addressed such as brain, heart, and lungs.^{41,42,43,45,47,48,49,50,51} Cardiovascular applications are uncommon in methods developed for anatomy segmentation and registration, mostly because of reduced availability of public datasets, small vessel sizes in the limbs, and large vessel occlusions due to severe stenoses.^52,53

3D Reconstruction and Visualization

3D reconstruction is the process of converting segmented imaging into a 3D file format, such as autodesk filmbox or object file. These files may then be loaded into a platform for 3D visualization, such as extended reality (XR), which consists of three modalities: virtual reality (VR), in which the user is fully immersed in a virtual space detached from reality; augmented reality (AR), in which virtual content is superimposed on the real world; and mixed reality (MR), in which virtual objects and real objects are able to interact with one another in some compelling capacity. Medical imaging may be combined with any of these modalities.

VR and AR enable target anatomies to be viewed in a more natural 3D context as opposed to a flat 2D monitor, which has been shown to enhance preoperative planning and surgical communication.^54,55 Surgeons using VR for preoperative planning have stated: “I felt like I had been there before,” pointing out how access to a virtual 3D representation of surgical anatomy helps improve the mental representation of plans, which in turn decreases the cognitive load of having to map 2D preoperative information to real-life 3D anatomy.⁵⁵ Furthermore, VR and AR can be used to enhance education, with most studies showing a positive association with the implementation of these technologies in medical education and learner outcomes.⁵⁶ Figure 2 shows a 3D model of a human heart as seen in a VR headset app. This model is generated by segmenting the heart in images obtained from a CT scan of a patient and creating a mesh that can be colorized and texturized. The user can “explore” the digital heart model by “manipulating” it using VR controllers.

Figure 2

The applications of MR to surgery are rapidly unfolding, with most US Food and Drug Administration (FDA) approvals granted after 2021.⁵⁷ Examples include Medivis’s Spine Navigation Platform, which allows physicians to overlay medical images with the patient’s body during surgery and MediView’s XR90, an AR system for visualization of organs and vasculature for needle-based soft tissue and bone procedures.

MR is most typically used to display segmented and registered patient imaging over the actual patient during surgery, enabling the surgical team to effectively “see through” the surgical field prior to incision and during minimally invasive surgery. This method of using MR has been applied successfully during minimally invasive spine surgery with FDA approval and applied preliminarily during congenital heart disease cases.^58,59 Figure 3 shows preliminary work using MR to visualize the patient body with a model of thoracic vasculature from a preoperative CT scan.

Figure 3

Surgical Robotics

The field of medical robotics has seen rapid growth, with thousands of robotic surgical systems installed in hospitals, performing thousands of procedures around the globe.^1,60,61,62 Robotic laparoscopic surgery is perhaps the most well-established application of medical robotics, partially due to the commercial success of Intuitive’s da Vinci robotic system (~8,600 da Vinci robots available on the market), with millions of procedures performed across multiple specialties.^1,8 The main advantage of its increased efficacy compared to manual laparoscopic techniques comes mostly from the increased tool dexterity offered by the robot wrists. Other benefits include improved surgeon ergonomics and the seamless integration with the endoscopic image and other imaging modalities in the surgeon’s view. Other commercially available robots for minimally invasive surgery include the Senhance^® Surgical System (Asensus Surgical), which prioritizes ease of use and efficiency,⁶³ and the Versius (CMR Surgical), which features an open console for easier communication with the surgical team.⁶⁴

In the cardiovascular domain, robotic systems mostly focus on the manipulation of catheters, guidewires, and other specialized tools that require higher precision for complex procedures. Additionally, radiation exposure minimization is critical given the high frequency of imaging that may be required from fluoroscopic devices in several vascular procedures where surgeons require navigation through vessels that may be hard to cross due to unknown types of lesions.⁵³ Examples of such robots include the Sensei X Robotic Catheter System, the CorPath GRX System,⁶⁵ and the Amigo Remote Catheter System⁶⁶ for cardiac electrophysiology, percutaneous coronary intervention, and peripheral vascular interventions.

Robotic-Assisted Image Acquisition

Another growing field in the use of medical robotics is robot-assisted medical imaging, where the robot’s main function is to facilitate the acquisition of medical images.⁶⁷ Most works to date on robot-assisted imaging are for US image acquisition.⁶⁸ To obtain US images, an operator is required to maneuver a probe against the human body and the quality of the image highly depends on the operator’s skills to maintain certain positions and forces on the desired anatomical space—making reproducibility a challenge. The introduction of robotic systems holds the promise to alleviate such challenges by augmenting precision and providing consistency. Additionally, robotic systems that position the US transducer may help reduce the number of musculoskeletal injuries among sonographers, reduce operator dependence, and attain more consistent 3D acquisition. An example of such a system enabled via telerobotics is currently being developed by Dopl Technologies, Inc.

Applications range from fully teleoperated and shared control to fully autonomous scanning.^{68,69,70,71,72,73,74,75} In the latter, AI methods such as modified U-Net architectures for automatic segmentation,⁷⁵ reinforcement learning agents for force control,⁷⁶ CNNs for relative motion prediction of the US probe,⁷⁷ and learning from demonstration for autonomous carotid artery scanning⁷⁸ have been proposed.

Another important image modality is X-ray. Robots for X-ray acquisition have been available for a few decades already with the widely known 4 DOF C-arms, which are robots that carry an X-ray source and detector at their end-effector. More recent C-arms include the Siemens Artis Pheno with 9 DOF. Fieselmann et al. developed a dual robot system where one arm has the source and the other one holds the detector, providing higher flexibility of image acquisition.⁷⁹ Pekel et al. show a 7 DOF robot arm that serves as a sample holder for CT scanning set up for non-standard trajectories such as spherical.⁸⁰ This could enable procedures with complex patient and operating room (OR) spatial configurations.

Reduction of ionizing radiation in patients and healthcare workers was explored by Guy et al. where a CNN is used for object detection to enable real-time tracking of a region of interest (ROI) to control a rapid lead shutter for collimating the X-ray beam to the ROI.⁸¹

Advances in Robot Autonomy

The continuous integration of imaging and robotics will allow an increase in robot autonomy, which is essential to attain the high levels of precision required in complicated operations such as spine surgery, partial nephrectomy, or lung biopsy. Most current preoperative planning already relies on some form of imaging, but planning is usually performed by the surgical team. High fidelity computational representations of patients’ anatomy and further availability of medical datasets for AI model training will enable automated surgical planning. Similarly, imaging is a key step during surgery since it provides a real-time estimate of the procedure’s current state, which is typically analyzed by the surgeon, who also decides a course of action.

Robot autonomy will increase surgery efficiency by reducing the frequency of handover between robot and surgeon, but it will require computational models capable of performing the steps referred to as active sensing, computational reasoning, and control. This has proven challenging for other domains of robotics, such as general navigation and manipulation, mostly due to the difficulty of the related computational problem. This may be particularly difficult for surgical robots where safety-critical behavior must also be guaranteed at all times. Additionally, autonomous surgical robots will need the ability to deal with the uncertainty that comes from the diversity of imaging modalities, patient anatomy, and unpredictable events while maintaining high efficacy.

Automatic processing of medical images is a necessary step to advancing robot autonomy. For example, the process of computing a volumetric mesh of anatomical parts from a patient CT usually requires human intervention to properly segment and edit the anatomy of interest to a state where it can be used for planning. Intraoperative CT (for example, cone beam CT) provides the opportunity to update models computed preoperatively due to patient positioning but requires the ability of high quality and fast autonomous image segmentation and 3D volume rendering. This process will improve with increasingly available and diverse datasets and with the advent of foundation models for medical images.

Examples of increased image-guided robot autonomy include fluorescence-guided autonomous robotic partial nephrectomy¹¹ with automatic tumor segmentation and robotics planning, point cloud-based registration of US to CT for automated robot-guided spine surgery,²⁰ and in vivo porcine experiments demonstrating autonomous needle navigation for lung biopsy under respiratory motion.⁸³ Other works consider the automation of general surgical skills such as robotic suturing⁸⁴ and knot tying.

Context-Aware Surgical Assistance

The revolution of surgical robotics will likely be driven by a close collaboration between the surgical team and physical AI systems, such as robotic systems powered by AI (computer vision, autonomous planning, machine learning, and control), which include surgical robotics, robots for image-acquisition, and robots for other tasks. A key concept is that of semantic understanding of surgical procedures,^{82,85,86,87,88} where computational models can automatically identify the stages of a specific procedure by tracking surgical instruments⁸⁹ and other stakeholders of surgery using a diversity of sensors. Such semantic understanding is key to providing pertinent context-aware surgical assistance to the surgeon.

Existing work in this direction has looked at robotic scrub nurses capable of anticipating surgical instruments on laparoscopic video⁹⁰ and robots that perform instrument handover based on video language models of surgery that integrate several state-of-the-art AI models.⁹¹ Future applications include intraoperative guidance from and towards the surgeon, early alert systems, improved communication among the surgical team, anomalies detection, enhanced training, and others.

Leveraging the models’ understanding of surgical procedures, robots for image acquisition will likely be partially or fully autonomous, facilitating the task of acquiring optimal imaging for a specific procedure and patient. For example, using cameras, depth and tracking sensors, and other devices, C-arms will be capable of autonomously finding optimal angles and views for fluoroscopic imaging during surgery without colliding with other robots and the surgical environment. Eventually, this will lead to the creation of intelligent ORs that can self-assemble based on specific patient anatomy and the surgeons’ inputs.

Surgeon-Robot Interaction

Before image-guided physical AI systems can be incorporated into the surgical pipeline, novel and efficient communication methods between them and the surgical team must be designed. The complexity of such collaboration will increase with further technological development and the increase of robot autonomy capabilities. Supervision and safe and efficient handovers will need to be considered for the wide variety of surgeries performed. This will likely require the use of multimodal communication for complex cases.

Current AI models, particularly those based on deep learning, often function as “black boxes,” offering limited transparency into how outputs are generated. This lack of interpretability poses significant challenges in medical environments, where surgeons, nurses, and other clinical staff must collaborate with robotic systems during highly complex procedures. To foster effective collaboration, AI-enabled surgical robots will require the integration of explainable AI—a research area that focuses on making certain parts of AI systems understandable by humans.⁹² By providing clear insights into their decision-making processes across all phases of surgery, these systems can enhance the surgical team’s understanding and support more informed, reliable decision-making.