ArticlePublisher preview available

SAGES video acquisition framework—analysis of available OR recording technologies by the SAGES AI task force

DOI:10.1007/s00464-022-09825-3
Authors:
Filippo Filicori
Filippo Filicori
Filippo Filicori
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Daniel P. Bitner
Daniel P. Bitner
Daniel P. Bitner
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Hans F. Fuchs
Hans F. Fuchs
Hans F. Fuchs
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Mehran Anvari at McMaster University
Mehran Anvari
Show all 12 authorsHide
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Background Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used for a variety of quality improvement, research, and educational applications. Various recording devices are available for standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kinematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we summarize available surgical video acquisition technologies and associated performance analysis platforms. Methods In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference. A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchronization of video with other data, availability of analytic tools, privacy, and access. Results Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%. Kinematic data are provided by 22% and perfusion imaging in one device. Conclusion Video acquisition platforms on the market allow for in depth performance analysis through manual and automated review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation, including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related to phase segmentation, instrument tracking, and video blurring.
Figures - available from: Surgical Endoscopy
This content is subject to copyright. Terms and conditions apply.
  • A preview of this full-text is provided by Springer Nature.
  • Learn more
Preview content only
Content available from Surgical Endoscopy
This content is subject to copyright. Terms and conditions apply.
Vol.:(0123456789)
1 3
Surgical Endoscopy
https://doi.org/10.1007/s00464-022-09825-3
ORIGINAL ARTICLE
SAGES video acquisition framework—analysis ofavailable
ORrecording technologies bytheSAGES AI task force
FilippoFilicori1,2· DanielP.Bitner1,2· HansF.Fuchs3· MehranAnvari4· GaneshSankaranaraynan5·
MatthewB.Bloom6· DanielA.Hashimoto7· AminMadani8· PietroMascagni9,10· ChristopherM.Schlachta11·
MarkTalamini2· OzananR.Meireles12
Received: 7 September 2022 / Accepted: 6 December 2022
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023
Abstract
Background Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used
for a variety of quality improvement, research, and educational applications. Various recording devices are available for
standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kin-
ematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis
through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for
automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we sum-
marize available surgical video acquisition technologies and associated performance analysis platforms.
Methods In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording
technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference.
A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchroniza-
tion of video with other data, availability of analytic tools, privacy, and access.
Results Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available
platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in
some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%.
Kinematic data are provided by 22% and perfusion imaging in one device.
Conclusion Video acquisition platforms on the market allow for in depth performance analysis through manual and automated
review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation,
including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related
to phase segmentation, instrument tracking, and video blurring.
Keywords Minimally invasive surgery· Surgical video· Operative recordings
The advent of minimally invasive surgery brought a large
increase in the availability of operative videos to a larger
extent than would have been anticipated when these
approaches were developed in 1980s and 1990s [1–3].
Increasing Audio–Video recording hardware translates to
more opportunities to record operative performance, and
logs of such video can serve for purposes of performance
review, quality analysis, research, and education. [Fig-
ure1] Prior work has outlined the medicolegal constraints
associated with intraoperative video, including data owner-
ship and storage, compliance with food and drug adminis-
tration (FDA) regulations, and privacy [4]. Generally, the
interests of several major parties (stakeholders) toward video
recordings can be outlined (Table1).
Aside from access to and security of the video, the con-
tent of the video and the tools that come with the associated
storage platform are important to both the surgeon and the
storage platform companies. Of interest, there are numerous
methods of editing and annotating the video, both manually
and automatically. Most manual performance assessment
tools come in the form of either lay (crowdsourced) or expert
surgeon annotations. Computer vision (CV), a field within
and Other Interventional Te
chniques
* Ozanan R. Meireles
OzMeireles@mgh.harvard.edu
Extended author information available on the last page of the article
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
ResearchGate has not been able to resolve any citations for this publication.
Global healthcare fairness: We should be sharing more, not less, data
Article
Full-text available
  • Oct 2022
The availability of large, deidentified health datasets has enabled significant innovation in using machine learning (ML) to better understand patients and their diseases. However, questions remain regarding the true privacy of this data, patient control over their data, and how we regulate data sharing in a way that that does not encumber progress or further potentiate biases for underrepresented populations. After reviewing the literature on potential reidentifications of patients in publicly available datasets, we argue that the cost—measured in terms of access to future medical innovations and clinical software—of slowing ML progress is too great to limit sharing data through large publicly available databases for concerns of imperfect data anonymization. This cost is especially great for developing countries where the barriers preventing inclusion in such databases will continue to rise, further excluding these populations and increasing existing biases that favor high-income countries. Preventing artificial intelligence’s progress towards precision medicine and sliding back to clinical practice dogma may pose a larger threat than concerns of potential patient reidentification within publicly available datasets. While the risk to patient privacy should be minimized, we believe this risk will never be zero, and society has to determine an acceptable risk threshold below which data sharing can occur—for the benefit of a global medical knowledge system.
View
Multicentric validation of EndoDigest: a computer vision platform for video documentation of the critical view of safety in laparoscopic cholecystectomy
Article
Full-text available
BackgroundA computer vision (CV) platform named EndoDigest was recently developed to facilitate the use of surgical videos. Specifically, EndoDigest automatically provides short video clips to effectively document the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). The aim of the present study is to validate EndoDigest on a multicentric dataset of LC videos.MethodsLC videos from 4 centers were manually annotated with the time of the cystic duct division and an assessment of CVS criteria. Incomplete recordings, bailout procedures and procedures with an intraoperative cholangiogram were excluded. EndoDigest leveraged predictions of deep learning models for workflow analysis in a rule-based inference system designed to estimate the time of the cystic duct division. Performance was assessed by computing the error in estimating the manually annotated time of the cystic duct division. To provide concise video documentation of CVS, EndoDigest extracted video clips showing the 2 min preceding and the 30 s following the predicted cystic duct division. The relevance of the documentation was evaluated by assessing CVS in automatically extracted 2.5-min-long video clips.Results144 of the 174 LC videos from 4 centers were analyzed. EndoDigest located the time of the cystic duct division with a mean error of 124.0 ± 270.6 s despite the use of fluorescent cholangiography in 27 procedures and great variations in surgical workflows across centers. The surgical evaluation found that 108 (75.0%) of the automatically extracted short video clips documented CVS effectively.Conclusions EndoDigest was robust enough to reliably locate the time of the cystic duct division and efficiently video document CVS despite the highly variable workflows. Training specifically on data from each center could improve results; however, this multicentric validation shows the potential for clinical translation of this surgical data science tool to efficiently document surgical safety.
View
Intellectual property and data ownership in the age of video recording in the operating room
Article
Full-text available
Background Recording intraoperative videos has become commonplace during surgery, with applications in video-based assessment, education and research. These videos can be both manually and automatically analyzed for performance analysis. A number of commercial entities providing video acquisition and processing has flourished over the recent years. As these companies expand, a number of medico-legal, licensing, intellectual property and data sharing related questions have been raised.Methods We performed a qualitative survey of surgeons, hospital administrators, lawyers and commercial entities offering video recording capabilities for serious issues that the average surgeon who records their videos should consider. To address these concerns, we reviewed relevant legal precedent and currently available contracts.ResultsWe identified several key medico-legal constraints, including data ownership and storage, FDA compliance, privacy and potential for use in litigation, present the legal background and potential solutions.Conclusion Given the availability of surgical recording and the future of video-based performance analysis, surgeons need to become comfortable with the medico-legal issues and the potential solutions available with national physician-led lobbying.
View
Video is better: why aren’t we using it? A mixed-methods study of the barriers to routine procedural video recording and case review
Article
Full-text available
IntroductionVideo-based case review for minimally invasive surgery is immensely valuable for education and quality improvement. Video review can improve technical performance, shorten the learning curve, disseminate new procedures, and improve learner satisfaction. Despite these advantages, it is underutilized in many institutions. So far, research has focused on the benefits of video, and there is relatively little information on barriers to routine utilization.MethodsA 36-question survey was developed on video-based case review and distributed to the SAGES email list. The survey included closed and open-ended questions. Numeric responses and Likert scales were compared with t-test; open-ended responses were reviewed qualitatively through rapid thematic analysis to identify themes and sub-themes.Results642 people responded to the survey for a response rate of 11%. 584 (91%) thought video would improve the quality of educational conferences. 435 qualitative responses on the value of video were analyzed, and benefits included (1) improved understanding, (2) increased objectivity, (3) better teaching, and (4) better audience engagement. Qualitative comments regarding specific barriers to recording and editing case video identified challenges at all stages of the process, from (1) the decision to record a case, (2) starting the recording in the OR, (3) transferring and storing files, and (4) editing the file. Each step had its own specific challenges.Conclusion Minimally invasive surgeons want to increase their utilization of video-based case review, but there are multiple practical challenges to overcome. Understanding these barriers is essential in order to increase use of video for education and quality improvement.
View
Formalizing video documentation of the Critical View of Safety in laparoscopic cholecystectomy: a step towards artificial intelligence assistance to improve surgical safety
Article
Full-text available
Background In laparoscopic cholecystectomy (LC), achievement of the Critical View of Safety (CVS) is commonly advocated to prevent bile duct injuries (BDI). However, BDI rates remain stable, probably due to inconsistent application or a poor understanding of CVS as well as unreliable reporting. Objective video reporting could serve for quality auditing and help generate consistent datasets for deep learning models aimed at intraoperative assistance. In this study, we develop and test a method to report CVS using videos. Method LC videos performed at our institution were retrieved and the video segments starting 60 s prior to the division of cystic structures were edited. Two independent reviewers assessed CVS using an adaptation of the doublet view 6-point scale and a novel binary method in which each criterion is considered either achieved or not. Feasibility to assess CVS in the edited video clips and inter-rater agreements were evaluated. Results CVS was attempted in 78 out of the 100 LC videos retrieved. CVS was assessable in 100% of the 60-s video clips. After mediation, CVS was achieved in 32/78(41.03%). Kappa scores of inter-rater agreements using the doublet view versus the binary assessment were as follows: 0.54 versus 0.75 for CVS achievement, 0.45 versus 0.62 for the dissection of the hepatocystic triangle, 0.36 versus 0.77 for the exposure of the lower part of the cystic plate, and 0.48 versus 0.79 for the 2 structures connected to the gallbladder. Conclusions The present study is the first to formalize a reproducible method for objective video reporting of CVS in LC. Minute-long video clips provide information on CVS and binary assessment yields a higher inter-rater agreement than previously used methods. These results offer an easy-to-implement strategy for objective video reporting of CVS, which could be used for quality auditing, scientific communication, and development of deep learning models for intraoperative guidance.
View
Surgical data science and artificial intelligence for surgical education
Article
  • Aug 2021
Surgical data science (SDS) aims to improve the quality of interventional healthcare and its value through the capture, organization, analysis, and modeling of procedural data. As data capture has increased and artificial intelligence (AI) has advanced, SDS can help to unlock augmented and automated coaching, feedback, assessment, and decision support in surgery. We review major concepts in SDS and AI as applied to surgical education and surgical oncology.
View
A Computer Vision Platform to Automatically Locate Critical Events in Surgical Videos: Documenting Safety in Laparoscopic Cholecystectomy
Article
Objective: To develop a computer vision platform to automatically locate critical events in surgical videos and provide short video clips documenting the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). Background: Intraoperative events are typically documented through operator-dictated reports that do not always translate the operative reality. Surgical videos provide complete information on surgical procedures, but the burden associated with storing and manually analyzing full-length videos has so far limited their effective use. Methods: A computer vision platform named EndoDigest was developed and used to analyze LC videos. The mean absolute error (MAE) of the platform in automatically locating the manually annotated time of the cystic duct division in full-length videos was assessed. The relevance of the automatically extracted short video clips was evaluated by calculating the percentage of video clips in which the CVS was assessable by surgeons. Results: 155 LC videos were analyzed: 55 of these videos were used to develop EndoDigest while the remaining 100 were used to test it. The time of the cystic duct division was automatically located with a MAE of 62.8 ± 130.4 seconds (1.95% of full-length video duration). CVS was assessable in 91% of the 2:30 minute long video clips automatically extracted from the considered test procedures. Conclusions: Deep learning models for workflow analysis can be used to reliably locate critical events in surgical videos and document CVS in LC. Further studies are needed to assess the clinical impact of surgical data science solutions for safer laparoscopic cholecystectomy.
View
Artificial Intelligence for Surgical Safety: Automatic Assessment of the Critical View of Safety in Laparoscopic Cholecystectomy Using Deep Learning
Article
Objective: To develop a deep learning model to automatically segment hepatocystic anatomy and assess the criteria defining the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). Background: Poor implementation and subjective interpretation of CVS contributes to the stable rates of bile duct injuries in LC. As CVS is assessed visually, this task can be automated by using computer vision, an area of artificial intelligence aimed at interpreting images. Methods: Still images from LC videos were annotated with CVS criteria and hepatocystic anatomy segmentation. A deep neural network comprising a segmentation model to highlight hepatocystic anatomy and a classification model to predict CVS criteria achievement was trained and tested using 5-fold cross validation. Intersection over union, average precision, and balanced accuracy were computed to evaluate the model performance versus the annotated ground truth. Results: A total of 2854 images from 201 LC videos were annotated and 402 images were further segmented. Mean intersection over union for segmentation was 66.6%. The model assessed the achievement of CVS criteria with a mean average precision and balanced accuracy of 71.9% and 71.4%, respectively. Conclusions: Deep learning algorithms can be trained to reliably segment hepatocystic anatomy and assess CVS criteria in still laparoscopic images. Surgical-technical partnerships should be encouraged to develop and evaluate deep learning models to improve surgical safety.
View
Association Between Surgeon Technical Skills and Patient Outcomes
Article
  • Aug 2020
Importance Postoperative complications remain common after surgery, but little is known about the extent of variation in operative technical skill and whether variation is associated with patient outcomes. Objectives To examine the (1) variation in technical skill scores of practicing surgeons, (2) association between technical skills and patient outcomes, and (3) amount of variation in patient outcomes explained by a surgeon’s technical skill. Design, Setting, and Participants In this quality improvement study, 17 practicing surgeons submitted a video of a laparoscopic right hemicolectomy that was then rated by at least 10 blinded peer surgeons and 2 expert raters. The association between surgeon technical skill scores and risk-adjusted outcomes was examined using data from the American College of Surgeons National Surgical Quality Improvement Program. The association between technical skill scores and outcomes was examined for colorectal procedures and noncolorectal procedures (ie, assessed on whether technical skills demonstrated during colectomy were associated with patient outcomes across other cases). In addition, the proportion of patient outcomes explained by technical skill scores was examined using robust regression techniques. The study was conducted from September 23, 2016, to February 10, 2018; data analysis was performed from November 2018 to January 2019. Exposures Colorectal and noncolorectal procedures. Main Outcomes and Measures Any complication, mortality, unplanned hospital readmission, unplanned reoperation related to principal procedure, surgical site infection, and death or serious morbidity. Results Of the 17 surgeons included in the study, 13 were men (76%). The participants had a range from 1 to 28 years in surgical practice (median, 11 years). Based on 10 or more reviewers per video and with a maximum quality score of 5, overall technical skill scores ranged from 2.8 to 4.6. From 2014 to 2016, study participants performed a total of 3063 procedures (1120 colectomies). Higher technical skill scores were significantly associated with lower rates of any complication (15.5% vs 20.6%, P = .03; Spearman rank-order correlation coefficient r = −0.54, P = .03), unplanned reoperation (4.7% vs 7.2%, P = .02; r = −0.60, P = .01), and a composite measure of death or serious morbidity (15.9% vs 21.4%, P = .02; r = −0.60, P = .01) following colectomy. Similar associations were found between colectomy technical skill scores and patient outcomes for all types of procedures performed by a surgeon. Overall, technical skill scores appeared to account for 25.8% of the variation in postcolectomy complication rates and 27.5% of the variation when including noncolectomy complication rates. Conclusions and Relevance The findings of this study suggest that there is wide variation in technical skill among practicing surgeons, accounting for more than 25% of the variation in patient outcomes. Higher colectomy technical skill scores appear to be associated with lower complication rates for colectomy and for all other procedures performed by a surgeon. Efforts to improve surgeon technical skills may result in better patient outcomes.
View
Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy
Article
  • Jul 2019
Objective(s): To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG). Background: Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identification of objects and patterns, such as in autonomous driving. Methods: Intraoperative video from LSG from an academic institution was annotated by 2 fellowship-trained, board-certified bariatric surgeons. Videos were segmented into the following steps: 1) port placement, 2) liver retraction, 3) liver biopsy, 4) gastrocolic ligament dissection, 5) stapling of the stomach, 6) bagging specimen, and 7) final inspection of staple line. Deep neural networks were used to analyze videos. Accuracy of operative step identification by the AI was determined by comparing to surgeon annotations. Results: Eighty-eight cases of LSG were analyzed. A random 70% sample of these clips was used to train the AI and 30% to test the AI's performance. Mean concordance correlation coefficient for human annotators was 0.862, suggesting excellent agreement. Mean (±SD) accuracy of the AI in identifying operative steps in the test set was 82% ± 4% with a maximum of 85.6%. Conclusions: AI can extract quantitative surgical data from video with 85.6% accuracy. This suggests operative video could be used as a quantitative data source for research in intraoperative clinical decision support, risk prediction, or outcomes studies.
View