Introduction
Robot-assisted minimally invasive surgery (RAMIS) has become increasingly prevalent for a range of procedures, including proctectomy, cholecystectomy, and hernia repair.1 The da Vinci system from Intuitive Surgical® is the most common RAMIS platform, installed in hospitals in more than 60 countries and used by over 55,000 trained surgeons.2,3 Advantages of RAMIS include 3-dimensional (3D) visualization, improved instrument manipulation, and reduced hospital stays.4,5 Despite these advantages, extensive training is needed to gain proficiency on these platforms.6 Training curricula typically require surgeons to perform to an acceptable degree, so assessing surgical proficiency is vital when implementing effective robotic surgery training programs. Surgical proficiency is typically characterized by technical skills (eg, instrument handling, collisions, and dexterity), and nontechnical skills (eg, procedural knowledge, stress management, and leadership). Proficiency in both of these categories is important for ensuring desired patient outcomes. Since nontechnical skills are difficult to measure, most assessment methods focus on differentiating based on technical skill scores.7
For open, minimally invasive, and robotic procedures, assessment of surgical skill traditionally relies on expert observation of trainee surgical performance. The most widely used assessment methods involve structured grading rubrics—for example, OSATS (Objective Structured Assessment of Technical Skill)—and procedure-specific checklists. These tools aim to provide objective evaluations of surgical ability based on clearly predefined criteria. Such assessments may be performed using physical models such as benchtop simulators, cadaveric specimens, or video recordings of clinical procedures. This diversity in task format allows expert evaluators to assess skills ranging from basic instrument handling to complex full-procedure performance. Despite their widespread use, structured assessments of surgical skill by experts have notable limitations. Primarily, they are time and resource intensive and may suffer from bias or inconsistencies in scoring.8
New methods of surgical skill assessment are emerging, particularly for RAMIS, that aim to reduce reliance on the expert evaluators typically required for structured grading assessments. Increasingly, these assessment techniques employ kinematic data from the robotic platform itself or other objective metrics that can be interpreted by nonexpert raters or machine learning models.9 Crowd-sourcing approaches have also been explored to lessen evaluator workload, using structured grading rubrics scored by large groups of nonexperts. Furthermore, the integration of robotic kinematic data with advanced computer vision and machine learning techniques is driving the development of automated surgical skill assessment systems, offering scalable alternatives to expert and crowd-sourced evaluation.10 To capture the latest developments in RAMIS technical skill assessment that are absent in less recent reviews, we surveyed articles published in the past 10 years, specifically focusing on studies using the da Vinci platform as their primary surgical system. First, we describe our search methodology used to gather the articles included in the survey. Next, we identify and explain the varying approaches to assessment represented in the survey articles. Finally, we discuss the applications and potential gaps in the current assessment literature.
Methods
Search Methodology
We used Scopus® to locate full text articles written in English that were published between 2014 and 2024. We used the query (robotic AND surgery) OR (robotic AND assisted AND surgery) AND (technical AND skill) OR (motor AND skill) AND (assessment).
Study Selection
The initial list of 189 results was screened for relevant articles and duplicates by reviewing the titles and abstracts of each article. The result was a list of 123 unique and potentially relevant articles. We reviewed the reduced list of articles to identify studies that covered topics related to the assessment of surgeons or surgical trainees while using one of the da Vinci robotic surgical platforms developed by Intuitive Surgical. This resulted in a list of 57 articles in the final analysis. Additionally, four papers focusing on automated skill assessment from the references of the curated list were added to create a final list of 61 papers (Figure 1).
Figure 1
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram shows selection process for papers included in final analysis.
Results
To understand the current landscape of assessment of robotic surgical performance, we conducted a review of current literature to identify grading approaches, methodologies, and trends and organized the approaches into categories based on the type of assessment used.
Structured Rubrics
Structured rubrics are assessment tools that consist of grading categories with clear criteria for evaluation. The areas assessed typically fall into two categories: technical skills and procedural knowledge. Common examples include Objective Structured Assessment of Technical Skills (OSATS),11 The Global Evaluative Assessment of Robotic Skill (GEARS),12 and the Global Operative Assessment of Laparoscopic Skills (GOALS).13 Because these rubrics have been validated to reliably distinguish between different levels of surgical expertise,14 they are widely regarded as the standard approaches to surgical skill assessment. As a result, structured grading rubrics are commonly used not only for trainee evaluation but also to validate surgical training curricula and emerging assessment methods.
OSATS is a structured rubric built to assess the technical skills of surgical procedures. Initially designed to train technical skills in the laboratory, OSATS has also been used as a performance assessment tool during operations.11 The OSATS rubric consists of seven categories that assess technical skills, such as respect for tissue, time and motion, use of assistants, handling and knowledge of instruments, flow of the operation, and knowledge of the specific procedure. These categories are assessed on a five-point scale, and the summed value of these individual ratings yields a Global Rating Score (GRS). Due to its established use and extensive validation, OSATS is routinely employed for assessing robotic procedures15,16,17,18,19,20,21,22 or for validating new assessment methods and curricula.23,24,25
GEARS is a grading rubric used to measure the surgeon’s technical performance when performing robotic surgery, evaluating skills in six areas: depth perception, bimanual dexterity, efficiency, force sensitivity, autonomy, and robotic control.12 Like OSATS, each area is assessed on a five-point scale. GEARS was designed to capture the nuances of operating on a robotic platform, which traditional rubrics like OSATS may overlook. Like OSATS, GEARS is often used to assess surgical procedure performance16,19,26,27,28,29 or validate other metrics and training curricula.30,31,32,33,34,35
GOALS is a structured grading method that assesses participants on depth perception, bimanual dexterity, efficiency, tissue handling, and autonomy, also based on a five-item global scoring rubric. GOALS has been shown to have high intraclass correlation values for surgeons, showing potential to discriminate between skill levels.13
Our survey also identified a number of alternative grading rubrics used to assess robotic surgery, many of which were developed to address specific tasks or limitations of more widely adopted tools such as OSATS and GEARS. These alternative rubrics offer diverse approaches to performance evaluation, often tailored to particular procedures or experimental settings.
A common method among researchers is to adapt standardized rubrics to better fit their specific use case. For example, a modified OSATS (mOSATS) is a rubric adapted from OSATS to include categories for suture handling and quality of the final product and excluding non-applicable categories.36 Other examples, such as robotic OSATS, rate depth perception, tissue handling, dexterity, and efficiency,37,38 while anastomosis OSATS, designed for gastrointestinal procedures, evaluates fifteen procedural steps on a three-point scale.39 Another example is the Structured Assessment of Robotic Microsurgical Skills (SARMS), which adapts the Structured Assessment of Microsurgical Skills (SAMS) rubric, with robotic-specific elements like wrist articulation and camera control.40 Modified GEARS is another example of a rubric designed to address gaps in the original by adding domains applicable to general robotic skills.41,42
Some rubrics propose new developments instead of simply modifying existing tools. The Assessment of Robotic Console Skills evaluates specific technical skill sets with six evaluation categories such as dexterity, field of view, and force sensitivity, showing reliability in distinguishing across surgeon skill levels.32 Another example is The Objective Clinical Human Reliability Analysis (OCHRA), an error-based tool that attempts to evaluate technical mistakes as the distinguishing factor in skill. Recent adaptations of OCHRA for basic robotic tasks demonstrate strong reliability and correlation with GEARS.35
Crowd-sourced Assessment Methods
A more recent approach to assessing surgical skills employs crowd-sourcing to generate rubric-based scores. Typically, expert surgeons administer grading rubrics; however, rubric scoring can be a time-intensive and tedious task, making it difficult to recruit experts willing to complete the assessments. Crowd-sourcing approaches solve this issue by having the performance rating done by a large group of laypeople.
Crowd-sourcing of surgical skill assessment is typically facilitated through platforms like Crowd-Sourced Assessment of Technical Skill (C-SATS), which recruits nonexpert raters, often via services like Amazon Mechanical Turk,43 to score surgical videos using structured grading rubrics such as GEARS in the case of C-SATS. These raters undergo brief training to familiarize them with the rubric criteria before evaluating video clips. Multiple independent ratings are then aggregated using statistical methods to produce reliable scores. This method has been shown to be comparable to expert rater scores when assessing generic technical skills37,44 and robotic radical prostatectomy.45
Skill-based Metrics
Surgical performance can also be assessed using individual skill-based metrics. These metrics are typically selected because they are inherently valuable in the surgical context (eg, patient outcomes, absence of intraoperative errors) or because they have been separately validated, often by comparison with established grading rubrics.
Completion Time
Time is a prevalent metric for robotic and traditional surgical performance. In the context of surgical assessment, time as a metric can be calculated as the time to complete an entire operation33,46 or individual tasks.15,20,47,48,49,50,51,52,53 Completion time has also been shown to be effective at distinguishing surgical skill level.15,16,53 This, paired with its ease of capture, makes completion a desirable and easy metric to include when performing skill assessment.
Kinematic-based Metrics
In robotic surgery, kinematic data refers to motion-related information from surgical instruments and robotic arms, such as position, velocity, acceleration, and joint angles captured from either the surgeon- or patient-side tools. Many robotic platforms are capable of recording this data during procedures; however, access is often limited due to proprietary restrictions.54 Despite this, when available, kinematic data allows researchers and system developers to design quantitative metrics for skill assessment and training.
A common approach to assess surgical performance in robotic surgery is to build metrics based on kinematic data collected from the surgical platform. Some of these metrics are based on tool velocity, acceleration, and other higher-order features that aim to capture the surgeon’s fine motor control and are used to distinguish technical skills. For example, higher mean tool speeds may indicate either improved efficiency or, conversely, a lack of control in less experienced users.55 These metrics can be sensitive to noise and therefore often require filtering or smoothing techniques to ensure robustness.51,55
Another frequently used metric is path length, the total distance traveled by the instrument tips during a task. In several studies, reductions in path length have been correlated with improved performance on structured rubrics, suggesting construct validity as a measure of surgical expertise.47,55,56
Completion-based Criteria
Researchers have also used task progression to evaluate performance, particularly in motor tasks performed on robotic platforms.15,57,58 This method involves breaking down a larger task into smaller sub-steps. Performance scores are typically reported in arbitrary units and are calculated based on how many sub-steps a participant completes. This metric has shown moderate to high correlations to GRS scores.15
Examples of other criteria include the completeness of tissue closure in suturing, the completeness or quality of tissue resection,46 or the requirement of additional resources or steps to complete the task.21
Outcome-based Metrics
When looking at outcome-based metrics, surgical skill or expertise is assessed based on the procedure’s outcomes. Outcomes include survival rates, intra/postoperative complications (eg, blood loss, tissue injury), and recovery indicators such as length of hospital stay or time to diet resumption.21,22,26,58 These outcome measures have been increasingly used to infer technical proficiency across diverse surgical contexts. Additionally, validated assessment tools like OSATS have shown significant associations with some surgical and postoperative complications.22
Robotic Control
User performance in robotic surgery can be evaluated based on how effectively robotic instruments are manipulated. Key metrics include maintaining instruments within the camera’s field of view,34 securely grasping or handling tissue and objects without dropping them,23,50 and avoiding collisions with tissue or other instruments.34,46 Instrument collisions are commonly measured by frequency, contact duration, or force and vibration levels during interaction.16,59
Studies have shown that experts tend to exhibit lower instrument vibration magnitudes, fewer collisions, and reduced contact forces compared to novices, all of which correlate with higher GRS and GEARS scores.16,46
Precision or Accuracy Errors
Precision or accuracy errors refer to deviations from ideal positions, orientations, or motion paths that characterize expert performance. These may result from poor robotic control or inadequate fine motor skills. Common metrics include needle entry/exit point deviation from predefined targets,57 instantaneous or root mean square (RMS) position and orientation errors,60 tool depth error along the viewing axis,50 and deviations in grasp position, orientation, or drive path (in or out of plane).59 Less common but relevant measures include missed targets, excess tissue piercings, and mechanical or unintentional suture failures.21,23,34,59
Others
Other metrics used to assess performance include the number of gripper activations,47 use of additional assist ports,61 camera use per minute, rate of orientation change, laterality,51 and other metrics related to knot security-knot tensile strength57,58 and the presence of air knots.23,34 It is important to note that many metrics—such as camera use per minute, rate of orientation change, laterality, normalized angular displacement—together with mean velocity and spectral arc length were not significantly associated with GEARS.51
In addition, researchers have used metrics that capture motion economy or efficiency by measuring normalized angular displacement,51 ribbon area, and console workspace volume.47,59
Other researchers have explored errors more broadly, as any of subjective importance noted by an expert surgeon or as error of omission or commission, with omission-based errors being those behaviors that have occurred partially or not at all and errors of commission being incorrect behaviors that have led to a definable mistake.62
Automated Assessment
Analyzing technical skills for robotic surgery in an automated manner can be done by developing machine learning (ML) or deep learning (DL) models that predict scores or skill levels in a supervised manner. Supervised ML/DL models need a training dataset consisting of signals that record relevant robotic tool motion, videos of the tasks being performed, and grading by experts, which serve as the ground truth. Skill assessment could be performed at the task, segment, or gesture level. One early work by Malpani et al. trained support vector machines on certain tool and camera motion feature data present in short segments to create a pairwise preference classifier, obtaining ≥ 85% accuracy when tested on video data from suturing and knot tying tasks.47
Most works have developed automated assessment models on the JHU-ISI Gesture and Skill-Assessment Working Set,63 popularly known as the JIGSAWS dataset. JIGSAWS includes synchronized videos and kinematic recordings of dry lab surgical tasks—specifically knot tying, needle passing, and suturing—along with self-reported skill levels and mOSATS scores. Data from the JIGSAWS dataset has been used to train models for predicting skill level and mOSATS scores.36,55,64,65,66 Skill level prediction, typically posed as a binary (novice/expert) or ternary (novice/intermediate/expert) classification problem, is judged in terms of percentage accuracy (%) in classification. The prediction of mOSATS scores is a regression problem and is typically judged in terms of the Spearman correlation metric (ρ), which is computed between the predicted scores and the expert scores on the test subset.
Prior to the advent of DL methods, research focused on using ML models such as k-nearest neighbors, logistic regression, or support vector machines to predict skill levels or scores36,55 and followed a more descriptive analysis approach; for example, they computed features or metrics from kinematic data and trained ML models on the features. Works that subsequently followed typically used raw kinematic tool motion data and trained expressive DL models such as 1-D Convolutional Neural Networks (CNNs)64,65 and CNNs with bidirectional long-short term memory (LSTM) models.66 These works have demonstrated high precision in predicting skill level.36,65 Fawaz et al. achieved 100% accuracy on suturing and needle passing tasks in the JIGSAWS dataset, while Benmansour et al. also showed Spearman correlation values with moderate-to-high correlation (ρ ≥ 0.6) when comparing predicted scores to OSATS scores.66
In addition to supervised approaches, one group was able to use self-supervised learning and uncertainty estimation to adapt a DL model that was originally trained on JIGSAWS data to a novel data set that consisted of videos showing participants performing the ring-transfer task in a virtual reality setting.60 The model was able to adapt without the need for manual labeling or task knowledge.
Due to the difficulties present in gathering proprietary kinematic data, some researchers have opted to assess skill through video alone. By using easily accessible video capture in combination with computer vision methods, researchers can estimate kinematic values and perform assessment using the estimates. However, this process can be challenging due to its multidimensional and complex nature. In a recent study by Funke et al.,67 3D CNN layers alongside a Temporal Segment Network were used to classify subject skill level (expert, intermediate, or novice) in the JIGSAWS dataset, achieving ≥ 95% accuracy in all three tasks.67 Lee et al. demonstrated that surgical tool tracking from robotic thoracic surgery videos enabled skill level prediction using OSATS and GEARS thresholds with 83% accuracy.68 Beyond just performing skill prediction, one group used surgical error annotations on Robot-Assisted Radical Prostatectomy (RARP) operations to train a deep learning model for automated error detection.69
Apart from using kinematic motion data or video data, Shafiei et al. asked participants to perform robotic tasks while wearing eye tracking devices and electroencephalogram (EEG) machines.70 Using generalized linear models, the authors demonstrated a link between EEG features, eye gaze features, and experience-related metrics with performance and learning rate evaluation measures, thus highlighting the potential of using auxiliary signals for skill assessment.
While it may be important to automate surgical skill assessment and scoring, it can also be critical to determine whether feedback from such methods can help users effectively learn surgical technical skills. Malpani et al.59 conducted a randomized controlled trial and demonstrated that automated real time teaching cues based on grasp orientation during virtual reality-based robotic surgical training are effective. Another recent randomized controlled trial focused on suturing tasks and feedback using artificial intelligence (AI) models.71 They showed that AI-based automated feedback—provided in the form of video clips demonstrating ideal performance—significantly improved needle handling scores, particularly for under-performers.
Discussion
While a variety of methods exist for assessing surgical training, structured rubrics remain the gold standard. Their widespread adoption is due to strong validation in distinguishing between different levels of surgical expertise as well as their established integration into formal surgical education programs. Rubrics such as OSATS and GEARS are not only used to assess trainee performance but have also become the benchmark for evaluating the effectiveness of alternative assessment techniques and the validity of training curricula. This significant role reinforces their influence in both clinical and research settings, enabling consistent comparisons across studies and institutions.
Most research has assessed surgical skill at a single level, procedure, task, or gesture without integrating these individual assessments into a full performance profile. While some researchers have compared metrics across various levels,56 they did not combine them into a full procedure assessment. This highlights an opportunity for future research to develop methods that aggregate lower-level assessments into larger comprehensive evaluations, potentially enabling assessments to be more robust, interpretable, and applicable to actual surgery.
Additionally, we noted that several studies in our survey employed surgical console simulators as platforms for performance assessment. These systems offer practical advantages in that they are low-cost, require no physical setup or expendable resources, and allow for consistent, repeatable task conditions across trials.20,24,29,31,49,51,72,73 However, results are mixed regarding their effectiveness for skill development. Some studies report no significant difference in performance outcomes between simulator-trained and traditionally trained participants73 while other studies suggest that simulator training may result in inferior performance compared to dry lab or cadaver-based training.72
There has also been a strong push towards automation in technical skill assessment, with eight out of thirteen automated skill assessment papers studied in this review published in the last 5 years. Several works have developed models using the JIGSAWS dataset.63 However, the JIGSAWS dataset has several limitations, including a small number of study participants and low-resolution video quality. Thus, there is a need for a large-scale skill assessment dataset that is openly available to the wider research community.
Researchers have also used crowd-sourcing for assessment and validation due to its cost efficiency and quick turnaround times. By implementing platforms such as C-SATS, researchers are able to perform assessments that are shown to correlate with expert raters assessment with lower costs and more quickly than traditional methods. Crowd-sourced grading costs approximately $16 USD per case compared to $50 to $100 USD for expert-based evaluations. This makes for a quick and affordable approach to performing surgical assessment. However, because crowd-sourced raters typically lack specific domain knowledge, much of the assessment is based on visual cues alone. This limits the use of crowd-sourced raters to applications where visual cues are sufficient to determine surgical performance, reducing the overall applicability of this method for medical programs.
Conclusion
Robotic surgical skill assessment has evolved significantly in recent years, with structured rubrics such as OSATS and GEARS remaining central due to their validation and widespread adoption. However, growing interest in objective, scalable, and automated methods has driven the development of skill-based metrics, ML models, and vision-based systems. While most studies assess isolated tasks or gestures, there is a clear need for multilevel frameworks that integrate fine-grained and procedural assessments into a unified understanding of surgical competence. As the field moves toward automation and personalized feedback, future efforts should focus on creating large, diverse datasets and cross-platform generalizable models to ensure accurate, efficient, and clinically meaningful evaluation of surgical performance.
Key Points
Structured rubrics (eg, the Objective Structured Assessment of Technical Skill and the Global Evaluative Assessment of Robotic Skill) remain the most validated and widely used methods but are limited by evaluator time burden and potential bias.
Crowd-sourced assessment platforms such as Crowd-Sourced Assessment of Technical Skill offer cost-effective and validated alternatives to traditional evaluation methods.
Automated assessment models using kinematic and video data show high accuracy in predicting skill level of surgeons, particularly with the use of deep learning approaches.
There is a need for assessment frameworks that integrate more granular assessment into larger performance profiles.
Competing Interests
The authors have no competing interests to declare.