-
[show abstract]
[hide abstract]
ABSTRACT: Background: Performance assessments rely on human judgment, and are vulnerable to rater effects (e.g. leniency or harshness). Making valid inferences from performance ratings for high-stakes decisions requires the management of rater effects. A simple method for detecting extreme raters that does not require sophisticated statistical knowledge or software has been developed as part of the quality assurance process for objective structured clinical examinations (OSCEs). We believe it is applicable to a range of examinations that rely on human raters. Methods: The method has three steps. First, extreme raters are identified by comparing individual rater means with the mean of all raters. A rater is deemed extreme if their mean was three standard deviations below (hawks) or above (doves) the overall mean. This criterion is adjustable. Second, the distribution of an extreme rater's scores was compared with the overall distribution for the station. This step mitigates a station effect. Third, the cohort of candidates seen by the rater is examined to ensure that any cohort effect is ruled out. Results and implications: Of 3000 + raters, fewer than 0.3% have been identified as being extreme using the proposed criteria. Rater performance is being monitored on a regular basis, and the impact of these raters on candidate results will be considered before results are finalised. Extreme raters are contacted by the organisation to review their rating style. If this intervention fails to modify the rater's scoring pattern, the rater is no longer invited back. As more data are collected the organisation will assess them to inform the development of approaches to improve extreme rater performance.
The Clinical Teacher 02/2013; 10(1):27-31.
-
Robyn Tamblyn,
Michal Abrahamowicz,
Dale Dauphinee,
Elizabeth Wenghofer,
André Jacques,
Daniel Klass, Sydney Smee,
Tewodros Eguale,
Nancy Winslade,
Nadyne Girard,
Ilona Bartman,
David L Buckeridge,
James A Hanley
[show abstract]
[hide abstract]
ABSTRACT: Less than 75% of people prescribed antihypertensive medication are still using treatment after 6 months. Physicians determine treatment, educate patients, manage side effects, and influence patient knowledge and motivation. Although physician communication ability likely influences persistence, little is known about the importance of medical management skills, even though these abilities can be enhanced through educational and practice interventions. The purpose of this study was to determine whether a physician's medical management and communication ability influence persistence with antihypertensive treatment.
This was a population-based study of 13,205 hypertensive patients who started antihypertensive medication prescribed by a cohort of 645 physicians entering practice in Quebec, Canada, between 1993 and 2007. Medical Council of Canada licensing examination scores were used to assess medical management and communication ability. Population-based prescription and medical services databases were used to assess starting therapy, treatment changes, comorbidity, and persistence with antihypertensive treatment in the first 6 months.
Within 6 months after starting treatment, 2926 patients (22.2%) had discontinued all antihypertensive medication. The risk of nonpersistence was reduced for patients who were treated by physicians with better medical management (odds ratio per 2-SD increase in score, 0.74; 95% confidence interval, 0.63-0.87) and communication (0.88; 0.78-1.00) ability and with early therapy changes (odds ratio, 0.45; 95% confidence interval, 0.37-0.54), more follow-up visits, and nondiuretics as the initial choice of therapy. Medical management ability was responsible for preventing 15.8% (95% confidence interval, 7.5%-23.3%) of nonpersistence.
Better clinical decision-making and data collection skills and early modifications in therapy improve persistence with antihypertensive therapy.
Archives of internal medicine 06/2010; 170(12):1064-72. · 11.46 Impact Factor
-
Elizabeth Wenghofer,
Daniel Klass,
Michal Abrahamowicz,
Dale Dauphinee,
André Jacques, Sydney Smee,
David Blackmore,
Nancy Winslade,
Kristen Reidel,
Ilona Bartman,
Robyn Tamblyn
[show abstract]
[hide abstract]
ABSTRACT: This study aimed to determine if national licensing examinations that measure medical knowledge (QE1) and clinical skills (QE2) predict the quality of care delivered by doctors in future practice.
Cohorts of doctors who took the Medical Council of Canada Qualifying Examinations Part I (QE1) and Part II (QE2) between 1993 and 1996 and subsequently entered practice in Ontario, Canada (n = 2420) were followed for their first 7-10 years in practice. The 208 of these doctors who were randomly selected for peer assessment of quality of care were studied. Main outcome measures included quality of care (acceptable/unacceptable) as assessed by doctor peer-examiners using a structured chart review and interview. Multivariate logistic regression was used to determine if qualifying examination scores predicted the outcome of the peer assessments while controlling for age, sex, training and specialty, and if the addition of the QE2 scores provided additional prediction of quality of care.
Fifteen (7.2%) of the 208 doctors assessed were considered to provide unacceptable quality of care. Doctors in the bottom quartile of QE1 scores had a greater than three-fold increase in the risk of an unacceptable quality-of-care assessment outcome (odds ratio [OR] 3.41, 95% confidence interval [CI] 1.14-10.22). Doctors in the bottom quartile of QE2 scores were also at higher risk of being assessed as providing unacceptable quality of care (OR 4.24, 95% CI 1.32-13.61). However, QE2 results provided no significant improvement in predicting peer assessment results over QE1 results (likelihood ratio test: chi(2) = 3.21, P-value((1 d.f.)) = 0.07).
Doctor scores on qualifying examinations are significant predictors of quality-of-care problems based on regulatory, practice-based peer assessment.
Medical Education 12/2009; 43(12):1166-73. · 3.18 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: The objective structured clinical examination (OSCE) requires the use of standardised patients (SPs). Recruitment of SPs can be challenging and factors assumed to be neutral may vary between SPs. On stations that are considered gender-neutral, either male or female SPs may be used. This may lead to an increase in measurement error. Prior studies on SP gender have often confounded gender with case.
The objective of this study was to assess whether a variation in SP gender on the same case resulted in a systematic difference in student scores.
At the University of Ottawa, 140 Year 3 medical students participated in a 10-station OSCE. Two physical examination stations were selected for study because they were perceived to be 'gender-neutral'. One station involved the physical examination of the back and the other of the lymphatic system. On each of the study stations, male and female SPs were randomly allocated.
There was no difference in mean scores on the back examination station for students with female (6.96/10.00) versus male (7.04/10.00) SPs (P = 0.713). However, scores on the lymphatic system examination station showed a significant difference, favouring students with female (8.30/10.00) versus male (7.41/10.00) SPs (P < 0.001). Results were not dependent on student gender.
The gender of the SP may significantly affect student performance in an undergraduate OSCE in a manner that appears to be unrelated to student gender. It would be prudent to use the same SP gender for the same case, even on seemingly gender-neutral stations.
Medical Education 06/2009; 43(6):521-5. · 3.18 Impact Factor
-
Robyn Tamblyn,
Michal Abrahamowicz,
Dale Dauphinee,
Elizabeth Wenghofer,
André Jacques,
Daniel Klass, Sydney Smee,
David Blackmore,
Nancy Winslade,
Nadyne Girard,
Roxane Du Berger,
Ilona Bartman,
David L Buckeridge,
James A Hanley
[show abstract]
[hide abstract]
ABSTRACT: Poor patient-physician communication increases the risk of patient complaints and malpractice claims. To address this problem, licensure assessment has been reformed in Canada and the United States, including a national standardized assessment of patient-physician communication and clinical history taking and examination skills.
To assess whether patient-physician communication examination scores in the clinical skills examination predicted future complaints in medical practice.
Cohort study of all 3424 physicians taking the Medical Council of Canada clinical skills examination between 1993 and 1996 who were licensed to practice in Ontario and/or Quebec. Participants were followed up until 2005, including the first 2 to 12 years of practice.
Patient complaints against study physicians that were filed with medical regulatory authorities in Ontario or Quebec and retained after investigation. Multivariate Poisson regression was used to estimate the relationship between complaint rate and scores on the clinical skills examination and traditional written examination. Scores are based on a standardized mean (SD) of 500 (100).
Overall, 1116 complaints were filed for 3424 physicians, and 696 complaints were retained after investigation. Of the physicians, 17.1% had at least 1 retained complaint, of which 81.9% were for communication or quality-of-care problems. Patient-physician communication scores for study physicians ranged from 31 to 723 (mean [SD], 510.9 [91.1]). A 2-SD decrease in communication score was associated with 1.17 more retained complaints per 100 physicians per year (relative risk [RR], 1.38; 95% confidence interval [CI], 1.18-1.61) and 1.20 more communication complaints per 100 practice-years (RR, 1.43; 95% CI, 1.15-1.77). After adjusting for the predictive ability of the clinical decision-making score in the traditional written examination, the patient-physician communication score in the clinical skills examination remained significantly predictive of retained complaints (likelihood ratio test, P < .001), with scores in the bottom quartile explaining an additional 9.2% (95% CI, 4.7%-13.1%) of complaints.
Scores achieved in patient-physician communication and clinical decision making on a national licensing examination predicted complaints to medical regulatory authorities.
JAMA The Journal of the American Medical Association 09/2007; 298(9):993-1001. · 30.03 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: The purpose of the current study was to assess the feasibility and validity of including a technical skill station on a national licensing examination. At the 2003 Medical Council of Canada Qualifying Examination, 745 test takers participated in a pilot station assessing the ability to perform a technical procedure. Checklists and rating scales were used for scoring. Validity was investigated by comparing surgery-trained to non-surgery-trained test takers. The mean for the pilot station was 72.4%. The pilot station was moderately correlated to the rest of the examination (item-total correlation .43). The mean score for surgery test takers was higher than for other test takers (P < .001). Inclusion of a technical skill station on a high-stakes examination is feasible, and at many levels, there is evidence of the validity of including this station.
American journal of surgery 01/2007; 193(1):86-9. · 2.36 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: The Medical Council of Canada (MCC) administers an objective structured clinical examination for licensure. Traditionally, physician examiners (PE) have evaluated these examinees. Recruitment of physicians is becoming more difficult. Determining if alternate scorers can be used is of increasing importance.
In 2003, the MCC ran a study using trained assessors (TA) simultaneously with PEs. Four examination centers and three history-taking stations were selected. Health care workers were recruited as the TAs.
A 3x2x4 mixed analyses of variance indicated no significant difference between scorers (F1,462=.01, p=.94). There were significant interaction effects, which were, localized to site 1/station 3, site 3/station 2, and site 4/station1. Pass/fail decisions would have misclassified 14.4-25.01% of examinees.
Trained assessors may be a valid alternative to PE for completing checklists in history-taking stations, but their role in completing global ratings is not supported by this study.
Academic Medicine 11/2005; 80(10 Suppl):S59-62. · 3.52 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: In 1994 and 1995, the Medical Council of Canada used an innovative approach to set the pass mark on its large scale, multi-center national OSCE which is designed to assess basic clinical and communication skills in physicians in Canada after 15 months of post-graduate medical training. The goal of this article is to describe the new approach and to present the experience with the method during its first two years of operation. The approach utilizes the global judgments of the physician examiners at each station to identify the candidates with borderline performances. The scores of the candidates whose performances are judged to be borderline are summed for each station, yielding an initial passing score for all stations and then the examination as a whole. The latter score is then adjusted upward one standard error of measurement for the final passing score and is used as one of the criteria to pass the examination. Based on the results to date, the new approach has worked well. The advantages, disadvantages and areas of possible refinement for the approach are reviewed.
Advances in Health Sciences Education 02/1997; 2(3):201-211. · 2.09 Impact Factor
