Machine-Learning Algorithm Highly Accurate for Automated Surgical Skill Assessment

Guido Beldi, M.D.

A machine-learning algorithm can help towards automation of surgical skill assessment.

The findings suggested that, after more testing, the AI could be implemented and effective in distinguishing good and poor surgical skill with high accuracy in clinical practice.

"The study is a first step. Now that we have demonstrated the fundamental feasibility, we can start planning assistance systems that will support surgeons during operations. For example, they will be alerted when fatigue is detected, thereby helping to prevent complications," lead study author Guido Beldi, M.D., said in a statement.

Beldi and colleagues aimed to automate surgical skill assessment in laparoscopic cholecystectomy videos using machine-learning algorithms. The video archive was screened for video recordings of laparoscopic cholecystectomies performed between January 2014 and May 2019. There was a total of 242 videos identified and they were segmented into procedural phases of the intervention.

Four board certified surgeons rated a total of 949 clip applications in 242 video recordings of laparoscopic cholecystectomy. Ratings were based on a Likert scale from one (minimum) to five (maximum).

Out of the 949 clip applications, 101 were randomly selected and partitioned into a training (60%), validation (20%), and testing split (20%).

The team used a three-step approach with AI. A convolutional neural network was trained to recognize the instruments. Then, the movements were analyzed, and their patterns were extracted. In the final step, the extracted movement patterns correlated with rating results by experts using linear regression.

A frame-wise instrument detection and localization model to predict the presence, type, and location of an instrument in each frame was trained. Clipper detection had an average precision of 78% and an average recall of 82%. The outputs from the detection and localization model were pre-processed before motion metrics were calculated. Pre-processing ensured individual instruments could be tracked throughout the clipping video segment.

Some of the clipper motion features demonstrated correlation with human rated skills. The motion features “Count” (P <.001), “Distance” (P <.001), “Radius 66%” (P <.001), “Radius 99%” (P <.001) and “Longest constant direction” (P <.001) were all negatively correlated with surgical skill ratings. The feature “Position change 1%” was positively correlated with surgical skill (P <.001). “Centroid x,” “Centroid y,” “Position change 10%,” and “Direction change” showed no significant correlation with the human rated skill ratings.

For the final step, the linear regression model was trained to predict surgical skill based on the extracted motion metrics. The model achieved a performance of 87±.2% in accuracy 1/0 and 70±.2% in accuracy +1/-1.

The study, “Automation of surgical skill assessment using a three-stage machine learning algorithm,” was published online in the journal Nature.