





































ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

QUALITY IMPROVEMENT OF 

AN AI SYSTEM FOR 

DETERMINING PASS-FAIL IN 

THE FUNDAMENTALS OF 

LAPAROSCOPIC SURGERY: 
ACCURACY ON A COHORT 

OF NEW USERS   

Finn Kliewer, Yiran Huang, Advaith Bongu, Yunzhe Xue, 
Andrew Hu, Usman Roshan   

 

✵ ABSTRACT 

Surgical residents must pass the 

fundamentals of laparoscopy surgery test to 

proceed with their training. While simulation gives 

them an environment to practice outside the 

operating room, it lacks supervision. To fill this gap, 

we recently proposed an AI system that evaluates 

pass-fail in the fundamentals of laparoscopic 

surgery. In this quality improvement study, we 

sought to evaluate the model’s accuracy when 

detecting a failure. To do this, we performed 

software testing on a cohort of high school students. 

The students were asked to conduct the essential 

peg transfer FLS task under the supervision of our AI 

system, which evaluates them in real-time as they 

perform the task. Out of 18 students, the system 

correctly predicted the student error in 13 cases. 

The model must catch up on student errors due to 

underlying model mispredictions in the remaining 

five. The model was not trained to handle such edge 

cases where it failed. These results show the 

potential of AI to make grading more fair, objective, 

and efficient. 

1 INTRODUCTION 

Surgical trainees preparing for a career in 

surgery must demonstrate proficiency in basic surgical 

skills. Evaluating these skills in the operating room 

requires considerable resources and increases the risk 

of surgical complications [1], [2]. Simulation has 

emerged as a cost-effective and accurate alternative [3–

6]. 

Laparoscopic surgery, also known as minimally 

invasive surgery, involves performing operations 

through small incisions with the assistance of a camera 

and specialized instruments. This approach offers 

benefits such as reduced pain, shorter recovery times, 

and decreased risk of infection compared to traditional 

open surgery. 

The Fundamentals of Laparoscopic Surgery 

(FLS) simulation kit is designed to train and evaluate 

students in laparoscopic surgery [7]. The first FLS task is 

to transfer six rings from one set of pegs to another 

set, picking up the ring with the left grasper, 

transferring it to the right grasper, and then placing it 

down on the goal peg. Once all rings are transferred to 

the right-side pegs, they must be moved back to the 

original left-side pegs. The requirements to pass are 

to make the transfer within 300 seconds and not drop a 

ring during the transfer. 

The FLS simulation kit replicates the conditions 

of laparoscopic surgery by placing a webcam within a 

non-transparent box. This setup mimics the lack of 

depth perception experienced when operating via a 

camera and screen, a typical challenge trainees face 

during laparoscopic procedures. Trainees must adapt 

to this limitation while using two graspers (Maryland 

dissectors) to manipulate the objects, forcing them to 

develop their hand-eye coordination and spatial 

awareness. 

Recently, our team proposed an AI system [8], 

[9] to evaluate automatically and grade students doing 

the peg transfer task using the FLS simulation kit. The 

current study builds upon this prior work, improving the 

existing model’s accuracy and robustness in providing 

standardized, objective evaluations. Grading these FLS 

exams are very time-intensive and subject to variation 

between graders, which can lead to evaluation 

inconsistencies. Some graders consider a particular 

performance a failure, while others may not. Using an 

AI grading system can standardize the process. The 

proposed AI system functions by analyzing the video 



ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

feed from the webcam inside the simulation kit. 

Once completed, the AI system generates a report 

displaying metrics for each move and an overall 

pass/fail score. In this quality improvement study, 

we evaluate our AI system on a cohort of medical 

high school students. Students were given verbal 

rudimentary directions on the FLS task beforehand 

and asked to conduct it while our software ran. While 

they carried out the task, the system evaluated their 

performance in real-time with overlayed feedback 

and looked out for any errors. When an error 

occurred, the system informed the student and 

stopped the evaluation. 

Of all 18 students who took the test, our 

system correctly detected errors in 13. In the 

remaining 5, the system encountered previously 

unseen scenarios, such as grasping a ring with both 

graspers, grabbing a ring in such a way that flung it 

out of bounds, and grabbing two rings 

simultaneously with one grasper. We expect the 

system to improve these predictions as we retrain 

with more data. 

 

2 METHODS 

OBJECTIVE 
In this study, our object was to test the 

accuracy of our model in detecting failures and 

errors during the peg transfer FLS task. 

Specifically, we sought to determine the accuracy 

with which our model could accurately flag these 

failure conditions. 

 

DATA 

We collected videos of 18 high school 

students, ages 16-18, performing FLS training on 

a school field trip at the Robert Wood Johnson 

Hospital in New Brunswick, New Jersey, USA. The 

aforementioned high school students had zero 

previous FLS training. Each student performed 

the task once, with collected videos ranging 

between 30 seconds to 2 minutes. 

Unexperienced high schoolers were perfect 

candidates for this study because we expected 

them to make many mistakes, which would test the 

limits of our system. All students who visited our 

booth during the field trip were included in the study, 

as there were no exclusion criteria. Since we were 

testing the AI’s ability to detect mistakes, we needed 

a group of students likely to make numerous and 

unpredictable errors. Below are screen snapshots of 

the FLS videos showing the left grasper picking a ring 

in (Figure 1(a)), a ring transfer between left and right 

graspers in (b), and placing a ring on the peg in (c). 

For each video, we determined a ground truth of 

whether a failure was present using the rules set by 

the medical team at Robert Wood Johnson Hospital. 

 

RULES FOR DETERMINING PASS OR FAIL 
1) Timing starts when the first ring is touched on the 

peg and the last ring is placed on the peg. Total 

time should not exceed 300s. 

2) The student fails the task if the ring is dropped 

outside of the FLS box. 

3) We look for six successful consecutive transfers 

from the left set of pegs to the right set. A single 

completed task is defined as grasping the ring 

with the left grasper, picking it from the peg, 

transferring it to the right grasper, grasping the 

ring with the right grasper, and finally placing it 

on the peg on the right side of the board. We 

summarize this as grasp1-pick-transfer-grasp2-

place. If six such transfers have occurred without 

a drop outside the box, continue to the next step. 

4) If a ring is dropped inside the box, it must be 

picked with the same grasper - in this case, 

grasper1. After picking up the correct grasper, the 

student can continue. However, if they pick with 

the wrong grasper, then this is a failure. 

5) We look for six successful transfers from the right 

set of pegs to the left set. We follow the same rules 

as we did above when transferring from left to 

right, except that a successful transfer is now given 

by grasp2-pick-transfer- grasp1-place. 

6) As not e d  above, if a ring d r o ps  inside the 

box, it must be picked with the same grasper - in 

this case, grasper2. After picking it up with the 

correct grasper, the student can continue, but 

picking with the wrong grasper is considered a 



ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

failure. 

7) If the six transfers have taken place without a 

drop outside the box and the total time is 

below 300 seconds, then the student has 

passed. 

 

 

(a) Pick a ring 
 

 
(b) Transfer ring 

 

 

 

 

 

 

 

 

 

 

(c) Place ring on peg 

 

Figure 1: Examples of picking a ring, transferring a 

ring, and placing one on the peg 

 

 Based on the above rules, we determined 

whether each video had a ground truth error. In 

Table I, we list the videos that we collected. 

 

 

AI SYSTEM 
Our overall approach is to pass a video 

through our trained YOLO (You Only Look Once) 

model, a machine learning model developed and fine-

tuned using a training dataset to efficiently identify 

objects in images or videos—in this case, rings and pegs. 

Our training dataset consisted of annotated video 

frames where the rings and pegs were marked. These 

annotations teach the YOLO model to recognize 

objects associated with our training data, such as rings 

and pegs. The size of our training data set was n = 

60 videos. The YOLO model recognizes objects by 

dividing the image into a grid and predicting bounding 

boxes and confidence scores for each grid section to 

help determine where our rings and pegs are. We then 

apply logic, which involves using predefined rules and 

conditions based on the task requirements, to analyze 

the model’s outputs. Specifically, we check whether the 

actions in the video follow the rules outlined above in 

Subsection II-C to determine if a resident passed or 

failed the FLS peg transfer task. See Figure 2 for an 

overview of our system. 

 

Figure 2: Overview of AI System 

 

3 RESULTS 

OVERALL METRICS 
In Table I below, we show our AI system’s 

ground truth and evaluation of each test video. Our 

model correctly identifies the failures in 13 of the 18 

videos recorded from the high school students’ field 

trip - thus giving an accuracy of 72.2%. In the three 

cases, it gives a wrong determination due to the 

system encountering an edge case it was not trained to 

handle (Figure 3) The other two mispredictions are 

due to incorrect bounding box predictions made by the 

underlying YOLO model (Figure 4). In Figure 4(a), the 

model incorrectly identifies two pegs as only one peg, 



ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

and in Figure 4(b), the ring dropped outside the box 

labeled as “drop in ring”. This procedure is expected 

to improve as we increase the training set size (the 

dataset used to train the machine learning model).  

4 DISCUSSION 
Out of 18 test runs, our model 

successfully detected errors in 13 instances. The 

five errors it failed to detect were unconventional 

and unlikely to be encountered in typical scenarios 

when surgical trainees perform the task. 

Unconventional errors refer to less systematic 

mistakes that go against the rules of the task, 

which are more likely to occur in individuals with 

no prior experience. This indicates that our AI 

model can effectively auto-grad the Fundamentals 

of Laparoscopic Surgery (FLS) exam. 
 

Video Failure Present Model Predicted Correctly 
1 Yes No 

2 Yes Yes 

3 Yes No 

4 Yes Yes 

5 Yes No 

6 Yes Yes 

7 Yes No 

8 Yes Yes 

9 Yes Yes 

10 Yes Yes 

11 Yes Yes 

12 Yes No 

13 Yes Yes 

14 Yes Yes 

15 Yes Yes 

16 Yes Yes 

17 Yes Yes 
18 Yes Yes 

Table 1: Medical resident level, ground truth, and 

prediction as given by the AI System 

 

(a) Ring being flung out of bounds 

 

 

 

(b) Picking up two rings with one grasper 

 

 

(c) Picking up a ring with two graspers 

 

Figure 3: Examples of the edge cases of the system was 

not trained to detect 

 

 

(a) Model draws one bounding box around two pegs 

 

 

(b) Model predicts a ring outside the box as “drop in 

ring” 



ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

 

Figure 4: Examples of the system not marking a 

failure due to incorrect bounding box predictions 

 

EDGE CASES 

In our testing, we encountered three 

unforeseen scenarios, including students grasping 

a ring with both graspers, pinching a ring leading 

to it being inadvertently flung out of bounds, or 

attempting to pick up two rings simultaneously with 

one grasper. We identified the cause of why the 

model incorrectly predicted each of these failures. 

These cases are considered edge cases since these 

failures typically do not happen when surgical 

trainees perform the task. In Figure 3(a), a ring flung 

out of bounds in the top left corner. The model 

did not catch this failure because the sample rate 

was too low and did not catch the ring flying out 

of bounds in time. We can fix this by adding a 

check to ensure all six rings are always on screen or 

increasing the sample rate. In Figure 3(b), we 

can see the grasper picking up two pegs 

simultaneously, and in Figure 3(c), two graspers 

picked up one ring. We did not anticipate such 

behavior when training the model. By retraining the 

system with additional data that includes these 

edge cases, as well as adding logic rules, we can 

improve its performance in identifying and marking 

such edge cases in future assessments. 

 

FUTURE IMPROVEMENTS 

Our system is the first to give a fully 

automated pass-or-fail outcome of the FLS task with 

real-time performative feedback. These scenarios 

are considered edge cases, which refer to rare 

or unexpected situations outside routine 

procedures. These cases are edge cases because 

these failures did not happen when the surgical 

trainees performed the task. To improve this 

system in the future, we will use the videos 

collected from this study to train the model in 

these specific edge cases. We will also add 

additional training data from resident surgeons to 

improve the underlying YOLO model and counter 

mispredictions. Another significant improvement on 

our agenda is adding more precise individualized 

directions and instructions overlayed on the screen to 

lower the failure rate. While this may not directly 

improve the accuracy of our model when predicting 

failures, this will help our system provide more 

effective training for resident surgeons when studying 

for the FLS exam. 

 

ETHICS 
Ethical concerns and potential future 

consequences must be addressed when considering 

implementing an AI system to grade FLS exams. The 

use of AI in this context raises questions about 

accountability; specifically, who is responsible for errors 

in grading—the researchers, the institution, or the AI 

system itself? While the AI system will provide significant 

time savings and standardize grading to eliminate 

variations between human graders, it is crucial to 

maintain transparency in grading decisions. It will be 

essential to monitor its impact on training and 

certification processes to ensure it enhances, rather 

than hinders, the development of skilled surgeons.  

5 REFERENCES 
 

[1] M. Bridges and D. L. Diamond, “The financial impact of 
teaching surgical residents in the operating room,” The 
American Journal of Surgery, vol. 177, no. 1, pp. 28–32, 
1999. 

[2] R. W. Allen, M. Pruitt, and K. M. Taaffe, “Effect of resident 
involvement on operative time and operating room 
staffing costs,” Journal of Surgical Education, vol. 73, no. 
6, pp. 979–985, 2016. 

[3] V. N. Palter and T. P. Grantcharov, “Individualized 
deliberate practice on a virtual reality simulator 
improves technical performance of surgical novices in 
the operating room,” Journal of the American College of 
Surgeons, vol. 213, no. 3, p. S126, 2011. 

[4] S. R. Dawe, G. Pena, J. A. Windsor, J. Broeders, P. C. 
Cregan, P. J. Hewett, and G. J. Maddern, “Systematic 
review of skills transfer after surgical simulation-based 
training,” Journal of British Surgery, vol. 101, no. 9, pp. 
1063–1076, 2014. 

[5] B. Zendejas, D. A. Cook, J. Bingener, M. Huebner, W. F. 
Dunn, M. G. Sarr, and D. R. Farley, “Simulation-based 
mastery learning improves patient outcomes in 
laparoscopic inguinal hernia repair: a randomized 



ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

controlled trial,” Annals of surgery, vol. 254, no. 3, 
pp. 502–511, 2011. 

[6] T. Cox, N. Seymour, and D. Stefanidis, “Moving the 
needle: simulation’s impact on patient outcomes,” 
Surgical Clinics, vol. 95, no. 4, pp. 827– 838, 2015. 

[7] N. J. Soper and G. M. Fried, “The fundamentals of 
laparoscopic surgery: its time has come,” Bull Am 
Coll Surg, vol. 93, no. 9, pp. 30–32, 2008. 

[8] Y. Xue, A. Hu, R. Muralidhar, J. W. Ady, A. Bongu, 
and U. Roshan, “An ai system for evaluating pass fail 
in fundamentals of laparoscopic surgery from live 
video in realtime with performative feedback,” in 
IEEE International Conference on Bioinformatics and 
Biomedicine (BIBM). IEEE, 2023, pp. 4167–4171. 

[9] Y. Xue, O. Eletta, J. W. Ady, N. M. Patel, A. Bongu, 
and U. Roshan, “A cascaded neural network system 
for rating student performance in surgical knot tying 
simulation,” in Proceedings of the 11th IEEE 
International Conference on Healthcare Informatics 
(ICHI), 2023. 

 
 

 
 

 
 
 
 
 
 
 

 
 
 

 
 
 
 

 

 
 
 
 
 
 

 
 

 
 
 
 
 

 
 
 
 

 
 
 

 
 
 
 
 

 
 

 
 



ARESTY RUTGERS UNDERGRADUATE RESEARCH JOURNAL, VOLUME I, ISSUE VI 

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 
 

 

Finn Kliewer graduated from Rutgers University in 2024 with a 

degree in Computer Science and a minor in Mathematics. During 

his time at Rutgers, he developed a strong interest in artificial 

intelligence and its real-world applications, especially in areas that 

combine technology and human impact. Outside of 

academics, Finn enjoys hiking, making music, and exploring 

creative projects that bring together his technical and artistic 

interests. He is excited to continue learning and working on 

projects that use AI to solve meaningful problems. 

 
Finn can be contacted at flk16@rutgers.edu. 

 

mailto:flk16@rutgers.edu.

	✵ Abstract
	1 Introduction
	2 Methods
	Objective
	Data
	Rules for Determining Pass or Fail
	AI System
	3 Results
	Overall Metrics
	4 Discussion
	Edge Cases
	Future Improvements
	Ethics
	5 References

