Figure 1 - uploaded by Jonathan A Cook
Content may be subject to copyright.
Source publication
The aim of this study was to assess whether a literature review of a technology can allow a learning curve to be quantified.
The literature for fiberoptic intubation was searched for studies reporting information relevant to the learning curve. The Cochrane Library, Medline, Embase, and Science Citation index were searched. Studies that reported a...
Citations
... The 105 adaptive RCTs identified in this study include diverse adaptive designs, such as GSD, SSR, Bayesian SSA, GSD + SSR, dropping a treatment arm, investigating both inferiority and superiority, adaptive enrichment and response-adaptive randomization. However, none of the 105 adaptive RCTs considered learning curve effects in the adaptive designs, although the learning process of a surgical procedure can critically impact the overall safety and effectiveness outcome assessment [8][9][10][11]. McCulloch et al. recommend a five-stage process to develop and evaluate a new surgical procedure, including idea, development, exploration, assessment and long-term study stages [34]. The learning curve of a new medical device and its associated procedure should be evaluated in the development and exploration stages before implementing an RCT to assess the effectiveness. ...
Aim
Adaptive designs are frequently used in drug randomized controlled trials (RCTs). However, their use in medical device RCTs remains unclear. We aimed to characterize medical device RCTs with adaptive designs.
Materials & methods
We searched for adaptive RCTs in the following databases: ClinicalTrials.gov, International Clinical Trials Registry Platform and the International Standard Randomised Controlled Trial Number registry. Adaptive design keywords and medical device corporation names were used as terms to search the trial records registered between 1 January 2000 and 18 October 2024 in the databases. The annual number and proportions of adaptive trials were analyzed, and characteristics such as design type, sponsor, therapeutic area, trial stage and regulatory status were summarized.
Results
Overall, 105 adaptive RCTs were identified from ClinicalTrials.gov, accounting for 2.112 per 1000 trials in 49,721 medical device clinical trials registered in ClinicalTrials.gov during the period. The average annual number of adaptive RCTs per 1000 clinical trials was the highest (8.55 ± 11.65) during 2005–2010, reduced to 3.33 ± 2.35 during 2011–2016, and significantly decreased to 1.29 ± 0.85 during 2017–2024 (p = 0.011). The most common adaptive designs were group sequential design (GSD, 50.5%), sample size reassessment (SSR, 17.1%) and investigating both superiority and non-inferiority (10.5%). Most RCTs were sponsored by the private sector (62.9%), conducted in Europe/North America (95.2%), in the field of heart disease (46.7%) and post-market trials (76.2%). Compared with pre-market RCTs, post-market RCTs showed more diverse adaptive designs such as response-adaptive randomization and adaptive enrichment.
Conclusion
The average annual proportions of adaptive medical device RCTs in ClinicalTrials.gov has reduced in the last 10 years. The most-used adaptive designs in medical device RCTs are GSD, SSR and investigating both superiority and non-inferiority.
... A classic learning curve is said to have three parts, an initial curve, an expert plateau and the decline curve. [4][5][6] In the initial curve, there is rapid step-by-step learning till the surgeon reaches a plateau phase. A plateau phase is one where the learning curve becomes static. ...
Purpose:
To elucidate the learning curve for posterior segment diagnostic endoscopy (DE) based on the results of a self-trained (ST) and a supervised (SUP) vitreoretinal surgeon.
Methods:
Retrospective review of medical records of DE performed between 2017 and 2023 by one ST and one SUP vitreoretinal surgeon at a tertiary eye care institute. Data were collected and the serial number of cases was plotted against the time taken for the procedure. A comparative regression plot was created for both the surgeons to know the slope of the learning curve. The start time was noted as that of attachment of the endoscope and the stop time was noted as the end of diagnostic evaluation. Procedures were divided into blocks of 10 cases each and the time taken for the procedures was calculated.
Results:
Total of 106 eyes (58 by ST surgeon and 48 by SUP surgeon) were included. For ST surgeon, the time taken for the surgery correlated inversely (reduced sequentially) with the serial number of the case till the 20th case (correlation coefficient = -0.5, p = .01), for SUP surgeon, the time taken for the surgery correlated inversely with the serial number of the case till the 10th case (correlation coefficient = -0.9, p = <0.0001) and then stabilized. Neither of the groups had any adverse events.
Conclusion:
About 20 cases for a self-trained and about 10 cases for a supervised vitreoretinal surgeon are required to get stable with DE. These observations have implications in creating a training module for DE with appropriate number of training cases.
... Smith et al. [17] have defined it as the number of procedures that must be performed for the outcomes to approach a long-term mean rate. Typically, an LC is characterized by an S-shaped curve with three stages: an early phase, during which new skill sets are acquired; a middle phase, in which the speed of learning rapidly increases; and an expert phase in which the performance reaches a plateau [49]. However, other curves have been proposed that involve a dip in the LC following the initial acceleration of the learning rate; this occurs especially with handling more challenging cases. ...
... The description of a surgeon's extensive prior experience is crucial for accurately quantifying the assessment of the learning curve, a point reported to be neglected during the assessment in various types of learning assessments related to healthcare procedures [49]. In our review, we observed the same conclusion in all included studies. ...
Background
The endoscopic endonasal transsphenoidal approach (EETA) has revolutionized skull-base surgery; however, it is associated with a steep learning curve (LC), necessitating additional attention from surgeons to ensure patient safety and surgical efficacy. The current literature is constrained by the small sample sizes of studies and their observational nature. This systematic review aims to evaluate the literature and identify strengths and weaknesses related to the assessment of EETA-LC.
Methods
A systematic review was conducted following the PRISMA guidelines. PubMed and Google Scholar were searched for clinical studies on EETA-LC using detailed search strategies, including pertinent keywords and Medical Subject Headings. The selection criteria included studies comparing the outcomes of skull-base surgeries involving pure EETA in the early and late stages of surgeons’ experience, studies that assessed the learning curve of at least one surgical parameter, and articles published in English.
Results
The systematic review identified 34 studies encompassing 5,648 patients published between 2002 and 2022, focusing on the EETA learning curve. Most studies were retrospective cohort designs (88%). Various patient assortment methods were noted, including group-based and case-based analyses. Statistical analyses included descriptive and comparative methods, along with regression analyses and curve modeling techniques. Pituitary adenoma (PA) being the most studied pathology (82%). Among the evaluated variables, improvements in outcomes across variables like EC, OT, postoperative CSF leak, and GTR. Overcoming the initial EETA learning curve was associated with sustained outcome improvements, with a median estimated case requirement of 32, ranging from 9 to 120 cases. These findings underscore the complexity of EETA-LC assessment and the importance of sustained outcome improvement as a marker of proficiency.
Conclusions
The review highlights the complexity of assessing the learning curve in EETA and underscores the need for standardized reporting and prospective studies to enhance the reliability of findings and guide clinical practice effectively.
... In the condition with remote-control operation, there was a difference in the time required to complete the task for the first time with and without the screen, but the reason for this could not be verified within the scope of this experiment. Thus, in this study, the learning curve depicting the improvement and convergence of the performance indices with experience, as shown by Cook et al. [35], was mirrored in the smart-speaker operation task. In particular, the learning curve in this study showed that the performance converged relatively early after the first operation, suggesting that overcoming the first operation may be important for learning the skill of smart-speaker operation. ...
Although commercial smart speakers are becoming increasingly popular, there is still much potential for investigation into their usability. In this study, we analyzed the usability of commercial smart speakers by focusing on the learnability of young users who are not yet familiar with voice user interface (VUI) operation. In the experiment, we conducted a task in which users repeatedly operated a smart speaker 10 times under four conditions, combining two experimental factors: the presence or absence of a screen on the smart speaker and the operation method (voice control only or in conjunction with remote-control operation). The usability of the smart speaker was analyzed in terms of task-completion time, task-completion rate, number of errors, subjective evaluation, and retrospective protocol analysis. In particular, we confirmed and compared the learning curves for each condition in terms of the performance metrics. The experimental results showed that there were no substantial differences in the learning curves between the presence and absence of a screen. In addition, the “lack of feedback” and “system response error” were identified as usability problems, and it was suggested that these problems led to “distrust of the system”.
... The following characterisation from Cook, Ramsay and Fayers [17] (pg. 255) appears to be the accepted foundation for conceptual models of the learning curve: "A learning curve...tends to be most rapid at first and then tails off over time. ...
Background
Surgical interventions are complex. Key elements of this complexity are the surgeon and their learning curve. They pose methodological challenges in the design, analysis and interpretation of surgical RCTs.
We identify, summarise, and critically examine current guidance about how to incorporate learning curves in the design and analysis of RCTs in surgery.
Examining current guidance
Current guidance presumes that randomisation must be between levels of just one treatment component, and that the evaluation of comparative effectiveness will be made via the average treatment effect (ATE). It considers how learning effects affect the ATE, and suggests solutions which seek to define the target population such that the ATE is a meaningful quantity to guide practice. We argue that these are solutions to a flawed formulation of the problem, and are inadequate for policymaking in this setting.
Reformulating the problem
The premise that surgical RCTs are limited to single-component comparisons, evaluated via the ATE, has skewed the methodological discussion. Forcing a multi-component intervention, such as surgery, into the framework of the conventional RCT design ignores its factorial nature.
We briefly discuss the multiphase optimisation strategy (MOST), which for a Stage 3 trial would endorse a factorial design. This would provide a wealth of information to inform nuanced policy but would likely be infeasible in this setting.
We discuss in more depth the benefits of targeting the ATE conditional on operating surgeon experience (CATE). The value of estimating the CATE for exploring learning effects has been previously recognised, but with discussion limited to analysis methods only. The robustness and precision of such analyses can be ensured via the trial design, and we argue that trial designs targeting CATE represent a clear gap in current guidance.
Conclusion
Trial designs that facilitate robust, precise estimation of the CATE would allow for more nuanced policymaking, leading to patient benefit. No such designs are currently forthcoming. Further research in trial design to facilitate the estimation of the CATE is needed.
... Our results show an overall improvement in the measured fine motor parameters as well as the time it took medical students to complete their tasks. These results indicate the effectiveness of the training course and the possible success of motion tracking with the inertial sensors used in our study, which were consistent with similar training simulator results [15,16]. Although all the parameters appear abstract, they are linked with the participant's performance in completing the tasks. ...
Background
Analysis of surgical instrument motion is applicable in surgical skill assessment and monitoring of the learning progress in laparoscopy. Current commercial instrument tracking technology (optical or electromagnetic) has specific limitations and is expensive. Therefore, in this study, we apply inexpensive, off-the-shelf inertial sensors to track laparoscopic instruments in a training scenario.
Methods
We calibrated two laparoscopic instruments to the inertial sensor and investigated its accuracy on a 3D-printed phantom. In a user study during a one-week laparoscopy training course with medical students and physicians, we then documented and compared the training effect in laparoscopic tasks on a commercially available laparoscopy trainer (Laparo Analytic, Laparo Medical Simulators, Wilcza, Poland) and the newly developed tracking setup.
Results
Eighteen participants (twelve medical students and six physicians) participated in the study. The student subgroup showed significantly poorer results for the count of swings (CS) and count of rotations (CR) at the beginning of the training compared to the physician subgroup (p = 0.012 and p = 0.042). After training, the student subgroup showed significant improvements in the rotatory angle sum, CS, and CR (p = 0.025, p = 0.004 and p = 0.024). After training, there were no significant differences between medical students and physicians. There was a strong correlation between the measured learning success (LS) from the data of our inertial measurement unit system (LSIMU) and the Laparo Analytic (LSLap) (Pearson’s r = 0.79).
Conclusion
In the current study, we observed a good and valid performance of inertial measurement units as a possible tool for instrument tracking and surgical skill assessment. Moreover, we conclude that the sensor can meaningfully examine the learning progress of medical students in an ex-vivo setting.
... The learning curve was assessed using the dissection speed and a performance score usually used in statistics to assess learning curves in laparoscopic surgery [24][25][26]. This score was considering the proper performance of the POEM and its complications. ...
... Nevertheless, the use of a similar performance score has already been used in a study on learning in an ex vivo model of colonic ESD. (25) It allowed for including in the analysis not only dissection speed, but also the absence of complications and the correct execution of the procedure based on the quality of myotomy. By using such assessment, we intended to provide a more robust picture of the operators' learning curve. ...
Background
Peroral endoscopic myotomy (POEM) is a very effective treatment for achalasia. However, training remains non-standardized. We evaluated a training curriculum, including ex vivo cases, followed by patients’ cases under expert supervision. The objective was to establish a learning curve of POEM.Materials and Methods
Four operators having completed advanced endoscopy fellowship were involved. They had already observed > 30 cases performed by experts. They performed 30 POEMs standardized (tunnel and myotomy lengths) procedures on ex vivo porcine model. Procedural times, number/volume of injections, mucosal and serous perforations, and myotomy length were collected. The learning curve was assessed using dissection speed (DS) and a dedicated performance score (PS), including learning rate (LR) and learning plateau (LP).ResultsThe operators completed all cases within 4 months (median of 3.5 cases/week). The mean procedural time was 43.3 min ± 14.4. Mean myotomy length was 70.0 mm ± 15.6 mm. Dissection speed averaged 1.78 mm/min ± 0.78. Using DS and PS as parameter, the LR was reached after 12.2 cases (DS = 2.0 mm/min) and 10.4 cases, respectively. When comparing the LP and the plateau phase, the DS was slower (1.3 ± 0.5 mm/min versus 2.1 ± 0.54 mm/min, p < 0.005) and perforations were decreased: 0.35 ± 0.82 in LP vs. 0.16 ± 0.44 in PP. Following this training, all operators performed 10 supervised cases and are competent in POEM.Conclusion
The association of observed cases and supervised ex vivo model training is effective for starting POEM on patients. The learning curve is 12 cases to reach a plateau.
... This relationship between learning effort and the outcome can be represented using learning curves (10,11). Factors that affect the learning curve are the initial skill level, the learning rate, and the final skill level achieved-known as the learning plateau (10,12,13). Understanding learning curves, both at individual and system levels, is crucial for assessing a new surgical technique or technology, informing surgical training, and evaluating procedures in practice (14,15). ...
... The "learning curve" is frequently used in surgical education literature and represents the relationship between learning effort and the outcome (11,12,15,31). Understanding the learning curve, rate, and plateau provides a mechanism for understanding the development of procedural competency (7,15). ...
Background
An exoscope heralds a new era of optics in surgery. However, there is limited quantitative evidence describing and comparing the learning curve.
Objectives
This study aimed to investigate the learning curve, plateau, and rate of novice surgeons using an Olympus ORBEYE exoscope compared to an operating microscope (Carl Zeiss OPMI PENTERO or KINEVO 900).
Methods
A preclinical, randomized, crossover, noninferiority trial assessed the performance of seventeen novice and seven expert surgeons completing the microsurgical grape dissection task “Star’s the limit.” A standardized star was drawn on a grape using a stencil with a 5 mm edge length. Participants cut the star and peeled the star-shaped skin off the grape with microscissors and forceps while minimizing damage to the grape flesh. Participants repeated the task 20 times consecutively for each optical device. Learning was assessed using model functions such as the Weibull function, and the cognitive workload was assessed with the NASA Task Load Index (NASA-TLX).
Results
Seventeen novice (male:female 12:5; median years of training 0.4 [0–2.8 years]) and six expert (male:female 4:2; median years of training 10 [8.9–24 years]) surgeons were recruited. “Star’s the limit” was validated using a performance score that gave a threshold of expert performance of 70 (0–100). The learning rate (ORBEYE −0.94 ± 0.37; microscope −1.30 ± 0.46) and learning plateau (ORBEYE 64.89 ± 8.81; microscope 65.93 ± 9.44) of the ORBEYE were significantly noninferior compared to those of the microscope group ( p = 0.009; p = 0.027, respectively). The cognitive workload on NASA-TLX was higher for the ORBEYE. Novices preferred the freedom of movement and ergonomics of the ORBEYE but preferred the visualization of the microscope.
Conclusions
This is the first study to quantify the ORBEYE learning curve and the first randomized controlled trial to compare the ORBEYE learning curve to that of the microscope. The plateau performance and learning rate of the ORBEYE are significantly noninferior to those of the microscope in a preclinical grape dissection task. This study also supports the ergonomics of the ORBEYE as reported in preliminary observational studies and highlights visualization as a focus for further development.
... An important aspect of measuring the LC is choosing the appropriate variables. There are two main types of variables: measuring the surgical process and measuring patient outcomes [6]. Measures of the surgical process include variables such as time to complete the procedure, blood loss, and the success or completion rate of the procedure, the conversion rate from laparoscopic/ robotic to open surgery, and so on. ...
... Terminology and checklist useful for reporting of learning curve no progress for > 15 min, intraoperative blood loss, blood transfusion, complications, mortality, 30-day/90-day mortality (applicable for hospital performance metrics), oncological resection (for cancers), recurrence of disease 5Measurement methods • Group splitting (ANOVA/multivariate logistic regression analysis in the presence of confounding variables) • CUmulative SUM (CUSUM) analysis • RA-CUSUM (risk-adjusted CUSUM) analysis (in the presence of case-mix bias) • Funnel plot (for comparing two learners) • Performance curve (for referential assessment)6 Stage of learning curve • Beginning (with respect to professional experience) • Learning (slow or steep) • Turning point (threshold/cut-off) • Plateau (after which there is no significant change in learning) • Inflection (negative deflection in learning curve due to complicated cases/emergency situation) 7Threshold case/cut-off • No. of cases to achieve stabilization of operative duration/minimizing complications or events/achieve proficiency 8Volume of surgery • Number of cases per year (applicable for certain oncological procedures where learning is dependent on volume of cases) 9Bias • Case-mix bias, learners' professional experience bias, any other selection bias10 Safety audit • Identification, mention, and rectification of adverse events (classify according to WHO standards) or complications (classify according to Clavien-Dindo classification) ...
Measuring the surgical learning curve (LC) has potential benefits for patient safety and surgical education. Even though it is known that the learning of a practical skill becomes easier with time, the LC remained underutilized in medicine and surgery for decades. The understanding of the LC amongst surgeons needs to be enhanced in terms of its components, measurement, application, and reporting. This prompted us to review the science behind the learning curve and enrich the existing knowledge and its application in surgical learning and research. Application of knowledge of learning curve can help the surgeons to more rapidly achieve/maintain a high expertise level, improvise outcomes for their patients, and facilitate the investigators to scientifically report, audit, or research the surgical learning/innovations.
... Research in health technologies has identified three main features of a learning curve that provide a descriptive contour of the curve. (a) the initial/starting level, capturing where the curve begins; (b) the rate of learning, capturing how quickly a particular level of learning is reached; (c) the asymptote/expert level, measuring the level at which learning stabilises (Cook et al., 2007). Other potential objective measures include number of trials/cases (or time required) for the user to learn, and rate of learning accuracy increase and variability decrease (Adi-Japha et al., 2008;Hoffman et al., 2018). ...
More recently, Explainable Artificial Intelligence (XAI) research has shifted to focus on a more pragmatic or naturalistic account of understanding, that is, whether the stakeholders understand the explanation. This point is especially important for research on evaluation methods for XAI systems. Thus, another direction where XAI research can benefit significantly from cognitive science and psychology research is ways to measure understanding of users, responses and attitudes. These measures can be used to quantify explanation quality and as feedback to the XAI system to improve the explanations. The current report aims to propose suitable metrics for evaluating XAI systems from the perspective of the cognitive states and processes of stakeholders. We elaborate on 7 dimensions, i.e., goodness, satisfaction, user understanding, curiosity & engagement, trust & reliance, controllability & interactivity, and learning curve & productivity, together with the recommended subjective and objective psychological measures. We then provide more details about how we can use the recommended measures to evaluate a visual classification XAI system according to the recommended cognitive metrics.
