Evaluating the spoken English proficiency of international medical graduates: detecting threats to the validity of standardised patient ratings.

Clinical Skills Assessment, Educational Commission for Foreign Medical Graduates, 3624 Market Street, Philadelphia, PA 19104, USA.
Medical Education (Impact Factor: 3.62). 02/2003; 37(1):69-76. DOI: 10.1046/j.1365-2923.2003.01400.x
Source: PubMed

ABSTRACT To investigate potential threats to the validity of the spoken English proficiency ratings provided by standardised patients (SPs) in high-stakes clinical skills examinations.
Spoken English ratings from 43 327 patient encounters were studied. These involved over 5000 candidates, 40% of whom were female and 33% of whom self-reported English to be their native language. Over 100 SPs were involved in the study, 51% of whom were female and 90% of whom were native English speakers. Possible performance differences in English ratings were studied as a function of candidate and SP gender, and as a function of candidate and SP native language (English versus all other languages).
No significant candidate by SP gender effect was detected. There were no meaningful differences in mean English ratings as a function of SP or candidate gender. Likewise, English ratings did not vary as a function of either candidate or SP native language. While candidate mean English ratings were not associated with the native language of the SP, native English-speaking candidates did achieve significantly higher ratings.
The lack of significant interaction between candidate and SP gender, and candidate and SP native language, suggests that the SPs provided unbiased English ratings. These results, combined with the expected higher English ratings given to candidates with English-speaking backgrounds, provides additional evidence to support the validity and fairness of spoken English proficiency ratings provided by standardised patients.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: U.S. surgery residency programs have traditionally attracted international medical graduates (IMGs). However, the qualifications and performance of IMGs are variable and difficult to predict. Poor performance negatively affects patient care, the residency program, and the IMGs. We sought to identify causes of poor performance and to develop a program to identify those with chances to succeed. Longitudinal study. Retrospective analysis. Description of a new program. University of Washington, a tertiary care teaching hospital. Performance of former IMG residents was reviewed to define the most common reasons for failure. In August 2002, we developed an IMG Certificate Program that enrolls IMGs into a formal 8-week clinical experience with duties, responsibilities, and evaluations similar to fourth-year medical students. A final global score is given for potential for success as a resident in our program. Poor performance in past IMG residents could be attributed to: credential problems and poor performance. Performance problems were further subdivided to include knowledge issues and personal/cultural issues. Since August 2002, our Certificate Program enrolled 15 IMGs. Fourteen graduated, and 10 were offered preliminary spots in our program: 4 are successful interns, 1 returned to Italy, and 5 will start in 2004. One entered the 2004 match in Anesthesiology, and 1 was counseled to not be a candidate for a U.S. program. Three had above average performance and were felt to be better suited to a smaller program (1-2 hospitals). The mean "potential for success" global score was 3.9 (all grads), 4.6 (current interns), 1.0 (nongraduate), and 3.0 for the above average performers better suited to a smaller U.S. program. We developed a program that provides IMGs an 8-week clinical experience in a busy U.S. training program; it provides them with enough experience to successfully integrate into a U.S. residency and identifies those with better chances to succeed. Wide application of this program and exchange of information among program directors may facilitate recruitment and the successful completion of training of IMGs and provide the number of residents needed to fill critical positions in the United States.
    Current Surgery 09/2004; 61(5):492-8. DOI:10.1016/j.cursur.2004.06.011
  • [Show abstract] [Hide abstract]
    ABSTRACT: The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise is specifically used to assess a candidate's ability to summarize and synthesize the data collected in a simulated patient interview. In a 1-year period, over 7700 first time takers completed the CSA, resulting in over 77,000 physician-based performance ratings. An initial pilot study indicated that, based solely on handwriting, the raters were able to correctly classify the gender of the candidate approximately 70% of the time. This result, combined with the fact that the notes are holistically scored, suggests that rating bias is possible. The purpose of this study was to investigate whether the gender of the candidate, the gender of the performing standardized patient, and the gender of the rater had any impact on scores. An analysis of covariance (ANCOVA) indicated that there was no significant interaction between candidate and rater gender. Female candidates significantly outperformed males, regardless of rater gender (p < 0.01, effect size = 0.23). The results of this study suggest that, based on rater, SP, and candidate characteristics, the validity of the PN ratings is not compromised.
    Advances in Health Sciences Education 03/2005; 10(1):53-63. DOI:10.1007/s10459-004-4297-y · 2.71 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Within health care, there have been and will continue to be many approaches to professional communication. Unfortunately, the body of evidence is very limited, and the research findings to support professional communication and the relationship with patient safety and quality are not available at this time. There were limited studies that tested specific interventions aimed at changing nurse-physician communication, and there is some evidence that focusing on a doctor-nurse communication may have a positive effect. Health care organizations and providers will be challenged as they seek to improve the effectiveness of professional communication, given all the subtleties of the nurse-physician relationships.
    Patient Safety and Quality: An Evidence-Based Handbook for Nurses, Edited by Ronda G Hughes, 04/2008: chapter Chapter 32; Agency for Healthcare Research and Quality (US).