Evaluating the spoken English proficiency of international medical graduates: Detecting threats to the validity of standardised patient ratings

Clinical Skills Assessment, Educational Commission for Foreign Medical Graduates, 3624 Market Street, Philadelphia, PA 19104, USA.
Medical Education (Impact Factor: 3.2). 02/2003; 37(1):69-76. DOI: 10.1046/j.1365-2923.2003.01400.x
Source: PubMed


To investigate potential threats to the validity of the spoken English proficiency ratings provided by standardised patients (SPs) in high-stakes clinical skills examinations.
Spoken English ratings from 43 327 patient encounters were studied. These involved over 5000 candidates, 40% of whom were female and 33% of whom self-reported English to be their native language. Over 100 SPs were involved in the study, 51% of whom were female and 90% of whom were native English speakers. Possible performance differences in English ratings were studied as a function of candidate and SP gender, and as a function of candidate and SP native language (English versus all other languages).
No significant candidate by SP gender effect was detected. There were no meaningful differences in mean English ratings as a function of SP or candidate gender. Likewise, English ratings did not vary as a function of either candidate or SP native language. While candidate mean English ratings were not associated with the native language of the SP, native English-speaking candidates did achieve significantly higher ratings.
The lack of significant interaction between candidate and SP gender, and candidate and SP native language, suggests that the SPs provided unbiased English ratings. These results, combined with the expected higher English ratings given to candidates with English-speaking backgrounds, provides additional evidence to support the validity and fairness of spoken English proficiency ratings provided by standardised patients.

7 Reads

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: U.S. surgery residency programs have traditionally attracted international medical graduates (IMGs). However, the qualifications and performance of IMGs are variable and difficult to predict. Poor performance negatively affects patient care, the residency program, and the IMGs. We sought to identify causes of poor performance and to develop a program to identify those with chances to succeed. Longitudinal study. Retrospective analysis. Description of a new program. University of Washington, a tertiary care teaching hospital. Performance of former IMG residents was reviewed to define the most common reasons for failure. In August 2002, we developed an IMG Certificate Program that enrolls IMGs into a formal 8-week clinical experience with duties, responsibilities, and evaluations similar to fourth-year medical students. A final global score is given for potential for success as a resident in our program. Poor performance in past IMG residents could be attributed to: credential problems and poor performance. Performance problems were further subdivided to include knowledge issues and personal/cultural issues. Since August 2002, our Certificate Program enrolled 15 IMGs. Fourteen graduated, and 10 were offered preliminary spots in our program: 4 are successful interns, 1 returned to Italy, and 5 will start in 2004. One entered the 2004 match in Anesthesiology, and 1 was counseled to not be a candidate for a U.S. program. Three had above average performance and were felt to be better suited to a smaller program (1-2 hospitals). The mean "potential for success" global score was 3.9 (all grads), 4.6 (current interns), 1.0 (nongraduate), and 3.0 for the above average performers better suited to a smaller U.S. program. We developed a program that provides IMGs an 8-week clinical experience in a busy U.S. training program; it provides them with enough experience to successfully integrate into a U.S. residency and identifies those with better chances to succeed. Wide application of this program and exchange of information among program directors may facilitate recruitment and the successful completion of training of IMGs and provide the number of residents needed to fill critical positions in the United States.
    Current Surgery 09/2004; 61(5):492-8. DOI:10.1016/j.cursur.2004.06.011
  • [Show abstract] [Hide abstract]
    ABSTRACT: To study possible differences in patient satisfaction ratings as a function of physician and patient ethnicity in a standardized patient (SP) performance-based assessment. Data from 334,397 ECFMG Clinical Skills Assessment (CSA) simulated clinical encounters were analyzed. A between-groups analysis of covariance was conducted, with independent variables consisting of SP and candidate ethnicity. Although a significant interaction between SP and candidate ethnicity was found, averaged over all encounters the SPs were equally satisfied with doctors from all cohorts. In general, SPs provided higher satisfaction ratings for racially concordant pairings. Among the ethnic groups of SPs, Asians were the least satisfied. Satisfaction can vary as a function of the ethnicity of the SP. These results are similar to studies of actual patient encounters that show Asians as less satisfied than patients from other ethnic backgrounds. However, for assessment purposes, provided there is a reasonable mix of SPs, the effect is quite small.
    Academic Medicine 11/2004; 79(10 Suppl):S15-7. DOI:10.1097/00001888-200410001-00005 · 2.93 Impact Factor
Show more