Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
Canadian Medical Education Journal
A critical analysis of The Royal College of Physicians and
Surgeons of Canada examination experience
Malcolm M MacFarlane
Correspondence to: Malcolm M. MacFarlane, c/o 22879-29B Avenue, Langley, BC, Canada, V2Z 3B1; email: m.macfarlane@nexicom.net
Published ahead of issue: Feb 17, 2025; CMEJ 2025 Available at https://doi.org/10.36834/cmej.79678
© 2025 MacFarlane; licensee Synergies Partners. This is an Open Journal Systems article distributed under the terms of the Creative Commons Attribution
License. (https://creativecommons.org/licenses/by-nc-nd/4.0) which permits unrestricted use, distribution, and reproduction in any medium, provided the
original work is cited.
Introduction
The Royal College of Physicians and Surgeons of Canada
(RCPSC) was established by an Act of Canadian parliament
in June 1929 to oversee post graduate medical education.1
Over the course of the past 95 years, the RCPSC has
established a prominent and highly influential role in
Canadian medical education and entry to practice. The
RCPSC influences the delivery of post graduate medical
education for every specialty except family medicine
(which is overseen by the College of Family Physicians of
Canada). Through their Competence by Design and
CanMEDS Framework,2 and through their accreditation of
all post graduate medical education programs in
Scientific Reports
Résumé
Résumé français à venir.
Abstract
The Royal College of Physicians and Surgeons of Canada (RCPSC)
plays a leading role in specialty and subspecialty post graduate
medical education (PGME) in Canada. As the RCPSC accredits
PGME programs, these programs are structured to meet the RCPSC
Competence by Design model and their CanMEDS roles. RCPSC
Certification is required by Medical Regulatory Agencies (MRAs)
across Canada as a condition of entry to independent practice. The
RCPSC relies heavily on the use of high-stakes subject examinations
as a key component of its Certification process. Recently, questions
have been raised regarding the usefulness of such high-stakes
examinations. If such examinations are to be fair and equitable,
they must be designed and implemented in accordance with best
practices for educational testing and the processes for
implementation and grading must be transparent and fair. This
paper reviews the recent literature on high-stakes examinations
and best practices in examination construction, references the
findings of a survey of RCPSC examination experiences conducted
by the Society for Canadians Studying Medicine Abroad exploring
the perception of respondents, and raises concerns regarding
RCPSC examinations related to validity, reliability, and fairness. The
paper concludes by recommending closer scrutiny of RCPSC
examination processes by interested stakeholders and by
provincial MRA's who delegate entry to practice decisions to the
RCPSC.
CANADIAN MEDICAL EDUCATION JOURNAL 2025
universities across Canada3 the RCPSC directs post
graduate specialty training in Canada. Almost every
provincial medical regulatory agency (MRA) in Canada
requires RCPSC Certification as a condition of full
Registration as a specialist. While the RCPSC is not
legislatively empowered to make entry to practice
decisions, most provincial MRAs have delegated their
legislative authority to make decisions about competence
for entry to practice in a specialty to the RCPSC as a third-
party assessor. As a result of this delegation of authority,
the RCPSC has assumed a leading role in determining entry
to practice to medicine in Canada, with RCPSC
examinations and Certification representing the final pre-
requisite to application for full licensing by a provincial
MRA. Given the significant role in entry to practice that the
RCPSC has come to play, it is important to critically examine
their Certification processes, and particularly their use of
specialty and subspecialty examinations, to ensure that
their examination processes are transparent, objective,
impartial and fair, as required by fairness legislation in most
provinces.
Standards for psychological and
educational testing
The RCPSC Certification process does not exist in a vacuum.
While the RCPSC is an independent not for profit Canadian
corporation with the mandate of overseeing post graduate
medical education, it exists within a legislative and
academic environment which demands certain standards
of practice for its Certification processes. An example of
one academic standard is that established by the Standards
for Educational and Psychological Testing (SEPT)
established jointly by the American Educational Research
Association, the American Psychological Association, and
the National Council on Measurement in Education.4
The SEPT are intended to provide guidelines for best
practices in developing and evaluating educational tests
and examinations and ensuring that educational tests and
their interpretation of test scores is valid for the test’s
intended uses. The SEPT provide guidelines for establishing
validity and reliability of educational testing. Validity refers
to the extent to which evidence and theory support that
test scores are consistent with their intended uses.
Reliability refers to “the consistency of scores across
replications of a testing procedure, regardless of how this
consistency is estimated or reported.”4(p33)
According to the SEPT, validity of a test may be evaluated
in several ways. One is content validity, or “an analysis of
the relationship between the content of a test and the
construct it is intended to measure.”4(p14) Another method
is convergent and divergent validity which involves
evidence of the extent to which test scores relate to, or
differ from, other measures intended to address the same
or different constructs. Reliability is typically assessed
through replications of the testing procedure to determine
whether test results are consistent over time. This may
involve retesting of the same individual over time,
determining whether two parts of a test measuring the
same criteria are consistent with each other (often referred
to as split-half reliability), and other psychometric
measures of consistency of test results.
In addition to concerns about validity and reliability, the
SEPT addresses the issue of fairness in testing. Tavakol and
colleagues in their paper on ensuring fairness in
assessment in health professions education emphasize the
important role the SEPT plays in ensuring that potential
biases in test development and administration are
minimized, and that tests are fair for all intended groups
regardless of examinee characteristics.5 According to the
SEPT, fairness includes elements such as fairness in
treatment during the testing process, fairness as a lack of
measurement bias, fairness in access to the constructs as
measured without the test taker being biased by personal
characteristics such as age, disability, gender, race,
ethnicity or language, which ensures that the test
measures only what it is intended to measure, and not
factors that are irrelevant. Fairness also includes ensuring
the validity of individual test score interpretations for the
intended uses. The SEPT note that when drawing
inferences about an examinee’s performance, skills and
abilities, it is important to consider how the examinee’s
individual characteristic such as ethnicity may interact with
the design and implementation of the testing situation.
They caution that, “Individuals who differ culturally and
linguistically from the majority of the test takers are at risk
for inaccurate score interpretations.”5(p53) This
consideration is particularly relevant when assessing the
competence of international medical graduates (IMGs).
The Medical Council of Canada
Qualifying Examination Part 1 as a
“gold standard”
The Medical Council of Canada Qualifying Examination 1
(MCCQE1) provides an excellent example of the above
SEPT standards in practice, and some might argue that it
represents a “gold standard” in assessment of medical
competency in Canada. The MCCQE1 is a standardized
CANADIAN MEDICAL EDUCATION JOURNAL 2025
examination that “assesses the critical medical knowledge
and clinical decision-making ability of a candidate at a level
expected of a medical student who is completing their
medical degree in Canada.”6 The MCCQE1 is required to be
completed by all IMG applicants to the Canadian Residency
Matching Service (CaRMS) for residency training, and is
part of the requirements for the Medical Council of Canada
Licentiate credential required by many provincial MRAs for
Registration of all applicants including Canadian Medical
Graduates (CMGs).
The Medical Council of Canada (MCC) regularly publishes a
document entitled “Annual Technical Report.”7 This
technical report transparently describes the MCCQE1
examination construction and outcomes with sections that
include:
• Exam development based on an exam blueprint
• Psychometric properties of the exam including:
o Standard setting
o Validity
o Reliability
• Exam security
• Quality assurance
• Examination Results including a breakdown of
cohorts into Canadian Medical Graduates (CMGs)
and International Medical Graduates (IMGs), and
first time and repeat test takers
• Results of a post examination candidate survey
The MCCQE Part 1 Annual Technical Report practices
correspond closely with the guidelines provided by the
SEPT. The MCCQE1 appears well researched with clear
transparent reporting of validity, reliability and
psychometric properties of the exam. It provides feedback
from candidates through a post-examination survey and
transparent reporting of results. This transparent reporting
creates confidence in the fairness of the MCCQE1
examination process which is also a key requirement of the
SEPT. Finally, it is worth noting that the MCC provides
examination candidates with a variety of exam preparation
resources,8 including practice tests, and that there are also
commercially available question banks which are a
valuable study aid in preparing for the examination.9
How does the RCPSC examination
process compare to the SEPT and
the MCC?
A search of the RCPSC website using search terms such as
“validity” and “reliability” reveals little information
regarding validity and reliability of their various specialty
exams. One document on the RCPSC website that relates
to the development of multiple-choice questions (MCQs)
for RCPSC exams is a 2004 publication that offers guidance
to committee members in developing MCQs.10 This
document describes the process for developing a well-
constructed MCQ and mentions the importance of reliable
and valid MCQs, but offers little information regarding how
to assess and document that MCQ exams are valid and
reliable. There is also a document on the RCPSC website
most recently edited in 2018 entitled “Roles and
Responsibilities of Examination Boards and Specialty
Committees with Regards to Examinations.”11 This
document describes a variety of reports available to these
committees including: a psychometric examination report,
a multiple-choice question report, and a post examination
survey report. This document contains strict limitations on
the extent to which information from these reports may be
shared, even within the RCPSC committees, and there
appears to be no public sharing of these reports permitted,
which reflects a concerning and apparently intentional lack
of transparency compared with the recommendations of
the SEPT and the practices of the MCC.
A search of the academic literature in Google Scholar using
the search terms such as “validity,” “reliability,” “RCPSC,”
and “examinations” also returned no studies researching
the validity or reliability of RCPSC exams. This dearth of
research may be a result of RCPSC confidentiality strictures
and reflects a troubling lack of transparency regarding
RCPSC exam processes. Regarding exam reporting, the only
score reporting found on the RCPSC website was a page
providing average pass rates of those who trained in
Canada by specialty or subspecialty.12 I was unable to find
any document comparable to the MCC Annual Technical
Report, and unlike the MCC Annual Technical Report, these
examination results were not broken down into cohorts
making it unclear how CMGs versus IMGs who trained in
Canada (or IMGs who trained outside Canada) performed
on this exam relative to each other.
CANADIAN MEDICAL EDUCATION JOURNAL 2025
Fairness and the RCPSC exams.
The SEPT document above describes critical elements
necessary for fair testing including fairness as a lack of
measurement bias and fairness in access to the constructs
as measured. Without a transparent and detailed technical
report such as MCC provides, and without transparent
access to psychometric data and reliability and validity
studies, it is difficult to assess how fair the RCPSC exams are
to different cohorts. In particular, the lack of data regarding
performance of CMGs relative to IMGs raises questions as
to whether there are differences between the
performance of IMGs relative to CMGs, and if so, the
origins of any differences.
As discussed below, some IMGs who have completed
RCPSC examinations have reported difficulty passing these
exams. These respondents attribute their difficulty passing
these exams to a variety of factors such as the inclusion of
Canadian specific minutiae in the exams that they deem
irrelevant to assessment of entry to practice levels of
competency, as well as lack of access to Canadian study
resources that CMGs and IMGs who trained in Canada have
ready access to from their programs. Without a
transparent report from the RCPSC regarding IMG
performance, it remains a question whether IMGs do more
poorly than CMGs on RCPSC exams, and we are forced to
rely upon anecdotal reports. Further, if indeed IMGs do
perform more poorly, we are unable to determine whether
this is due to CMGs being more competent than IMGs, or
whether this is due to other factors including failure to
design an examination that meets SEPT standards of fairly
assessing candidates from different cultural and linguistic
backgrounds, thus providing CMGs and IMGs trained in
Canada with an unfair advantage.
Prat, in Healthy Debate,13 describes what he experienced
as a lack of fairness and transparency from the RCPSC
regarding his request for a formal review of his exam scores
based on a belief that he failed the psychiatry specialty
exam due to factually inaccurate questions being included
in the exam. Prat notes that, “The examination process and
appeal do not permit anyone to review the content of the
examination as it is deemed ‘confidential to the Royal
College and not shared with candidates.’ There is therefore
no possibility for the candidates to check the accuracy of
the expected answers, not even for the sake of
understanding where they failed.” Prat goes on to state
that, “When a request for appeal is made, there is limited
chance of success since candidates do not have access to
anything tangible. Moreover, the Royal College does not
accept appeal requests for what it considers to be an
‘alleged error in content.’”
This RCPSC policy precluding an appeal based on questions
regarding inaccurate content is described in the RCPSC
Policy on Formal Review of Examinations,14 which states,
Formal reviews of examinations are conducted only
based on alleged significant procedural irregularities
in the assessment process, not because of alleged
errors in content. The exclusion of errors in content
applies to allegations of errors in either the questions
and structure of the examination, or the evaluation
and content of the responses provided by the
candidate.
Prat concludes that, “The Royal College examination
process is not transparent,” and that, “A fair and
transparent process needs to be in place.”
Fairness and the SOCASMA RCPSC exam experiences
survey.
Another indication of public and candidate perceptions
regarding the RCPSC examination processes may be
derived from a reading of the SOCASMA RCPSC
Examination Experiences Survey, a survey undertaken by
the Society for Canadians Studying Abroad (SOCASMA), a
summary of which is posted on their website.15 This Survey
was completed between December 2022 and February
2023. SOCASMA surveyed fifty-three anonymous
participants who completed an online Google Forms survey
about their experiences with a variety of RCPSC specialty
exams. As their research referenced here relies exclusively
on secondary use of anonymous information it does not
require Research Ethics Board approval to be reproduced
in this paper (see Article 2.4).16
Caution needs to be used in interpreting the results of the
SOCASMA RCPSC Examination Experiences Survey. The
number of respondents is low with only fifty-three
respondents. The summary posted online contains only a
brief description of the methodology of the survey. It states
that “participants were recruited though invitations and
notices on various medical forums and through interested
stakeholder groups,” and goes on to caution that, “this
survey may be subject to some degree of response
bias.”16(p1) Given the reported methodology of the survey,
selection bias is likely. The combination of selection bias
and low respondent numbers limit the generalizability of
the survey. Also, based on the online summary, the authors
conducted only a surface analysis of quantitative data. As
an example, and as noted later in this paper, there is no
reporting of how many CMGs vs IMGs had access to past
CANADIAN MEDICAL EDUCATION JOURNAL 2025
MCQ questions, and no analysis of how such access may
have impacted reported respondent pass rates on RCPCS
exams. Further, the study relies entirely on self-reported
data. The survey included considerable anecdotal
qualitative data, which is subjective in nature and not
subject to independent verification. Despite these
limitations, in the absence of any transparent reporting by
RCPSC of the results from their post examination survey
reports, this survey appears to represent the only publicly
available information regarding RCPSC examination
participant’s experience of the RCPSC examination process.
As such, the Survey feedback is worth considering.
In terms of Survey demographics, the RCPSC Examination
Experiences Survey reported that “67.3% of respondents
were International Medical Graduates (IMGs), 30.8% were
Canadian Medical Graduates (CMGs) and 1.9% were US
Medical Graduates (USMGs).”15(p1) One of the questions
the Survey asks respondents is, “In your opinion, do you
believe the RCPSC exam you wrote was objective and fair?”
Of 33 responses, 18 responded “Yes” and fifteen
responded “No” resulting in 45.5% of respondents not
believing that the RCPSC exam they wrote was objective
and fair. This suggests a significant perception problem
with the objectivity and fairness of RCPSC exams in the
population sampled. This is concerning given that the
Survey also reports that 34 of the 53 respondents or 64%
of the sample reported passing their RCPSC exams,
suggesting that respondents may not be simply
complaining about exams they have failed. Again, caution
should be used in interpreting these results given that the
survey does not appear to have conducted any cross
tabulations to determine how the respondents’ answers
regarding whether the exam was objective and fair
intersected with their having passed or failed their exam.
Another question asked in the Survey is, “When you think
about the overall exam experience, do you believe it was
fair and transparent?” Of 38 responses to this question, 18
responded “Yes” and 20 responded “No” resulting in 52.6%
of respondents thinking the overall exam experience was
not fair and transparent. As noted above, this data should
be interpreted with caution given the methodological
issues of the survey and the absence of reporting of any
deeper data analysis. For perspective on the above data,
while the MCCQE1 Candidate Survey in their 2020
Technical Report7 does not ask specifically for candidate’s
experiences of exam fairness, there are several questions
that are analogous and may offer points of comparison.
The MCCQE1 Candidate survey asks respondents, “How
would you rate your overall exam experience?(p44). For the
January 2021 examination cohort, 84% of candidates rated
their exam experience as Good, Very Good, or Excellent.”
The Candidate Survey also asks if, “The MCQ section
provided an opportunity for me to demonstrate my level of
medical knowledge.”(p42) For the January 2021 examination
cohort, 78% of candidates agreed or strongly agreed with
this statement, with a further 18% neither agreeing nor
disagreeing. Another question that speaks to MCCQE1
perceptions of fairness is, “The questions were clearly
written.”(p42) For the January 2021 examination cohort,
74% agreed or strongly agreed with this statement, with a
further 17% neither agreeing nor disagreeing. Given the
above MCCQE1 responses, reports that 52.6% of
respondents do not think the overall RCPSC exam
experience was fair and transparent appears to be a
concerningly high percentage. It is also concerning that the
RCPSC appears to conduct a candidate survey following
examinations, but that they choose not to share the results
of these surveys publicly resulting in a lack of transparency
regarding how candidates view the various RCPSC
examinations.
Respondents had an opportunity to provide comments
regarding why they answered the above questions on the
RCPSC Examination Experiences Survey the way they did.
These qualitative responses are quoted below in Table 1,15
and provide some insight into areas of concern regarding
the RCPSC exam process.
From the Table below, it is apparent that, at least based on
the Survey sample, there is a candidate perception that
RCPSC exams are not transparent, objective, or fair. Some
of the apparent themes include a perception that a
considerable number of questions are focused on minutiae
rather than on assessing core competency for entering
practice. From another Survey question, 55.5% of
respondents did not believe the questions encountered on
RCPSC exams were relevant and appropriate to assessing
readiness to enter independent practice. Other issues
include a perceived lack of transparency regarding how
MCQ questions are created and, consistent with Prat’s
comments, a perceived lack of transparency and candidate
trust and confidence regarding how exams are scored and
how potentially incorrect information in exam questions
are dealt with by the RCPSC. According to the Survey, 17.2
% of respondents reported that 11% to 40% of questions
contained factual errors, suggesting that concerns
regarding test construction are a significant issue.
CANADIAN MEDICAL EDUCATION JOURNAL 2025
Table 1. Qualitative survey comments
Feedback
“The questions were focused on minute details of subspecialties rather than general surgery questions that speak to safety to practice.”
“Some questions do not have a clear answer, do not reflect real clinical situations, no references found after the exam to support one particular answer.”
“I had some old exams but not the most recent couple years. The Canadians I met when I moved here had every question from every year which made the exam
a cake walk for them.”
“Some questions were narrowly focused and based on single studies which at times appeared arbitrary.”
“The exam contained a lot of new data that is not relevant to a newly graduated non sub specialized surgeon. Also they asked about Hazard ratio. who memorize
hazard ratio in trials!”
“Questions were very poorly written, in some cases clearly had been translated from French as the grammar was French (word order). Multiple answers were
often correct and clinically inappropriate to choose one over another given that both interventions/treatments are critical e.g. fluids, epinephrine etc.”
“Many of the questions are apparently repeats that certain individuals and programs have access to for practice and comprises a majority of the questions on the
exam. There were also many subspecialty questions not applicable to general practice”
“Canadian students/residents had access to questions well in advance and the ability to ask their supervisors (the ones writing the questions) what the answers
were. These answers were shared around to Canadians. Speaking to colleagues who were CMGs and passed the exams well after the fact, they pretty uniformly
say there’s no way anyone can pass that exam without the previous questions and specific coaching. One told me that even the course doesn’t really help (I took
it), you need the past questions.”
“The RC do not provide the answers that are expected, so there is no way to know if they have made a mistake or not. We need to blindly rely on the knowledge
content expert with no way to double checking. There is lack of transparency in their process to design the questions and no certainty that they use scientific
references to design the questions.”
“No transparency at all. We do not know how the questions are constructed or reviewed.”
“Many exam questions focused on minutiae rarely encountered in actual practice, high degree of subjectivity with oral assessment.”
“The Canadian residents have access to a resource (past questions) that International students do not.”
“I have been told by numerous people who have taken the exam (CMGs and IMGs) that there is no way to pass the exam without having access to illegally shared
databanks of remembered past exam questions, and that the Royal College recycles up to 80% of past exam questions, that they are overly detailed and lacking
in clinical relevance and that simply studying will not be enough to pass. It feels like competing in the Olympics when it’s an open secret that everyone is taking
steroids, and if you don’t take drugs you will not be able to compete/succeed.”
"Canadian trainees have the old exam questions and the exam is almost entirely old questions. If you don’t have some way to get them, it would be very difficult
to pass the exam, almost impossible. Additionally, there are no standardized/accepted materials such as books or question banks to study from as are available in
other countries. In the USA, there are books that are known to be the standard for studying, there are question banks provided by the board that are excellent
preparation. The exam was poorly written and many questions focus on extremely subspecialty information that you would never manage
In contrast to the above candidate perceptions, the RCPSC
describes a rigorous process for developing MCQ
examination questions.17 However, despite these rigorous
guidelines, the RCPSC does not transparently disclose any
information from their psychometric analysis of concluded
exams including information regarding number of
questions discarded for statistical or psychometric reasons,
a portion of which may have contained factual errors. It is
not unusual that formal review processes for high-stakes
exams do not include a review of factual errors in the
questions asked. This appears to be the case for
reconsiderations of the MCCQE1 which only conducts a
reconsideration due to exam day incidents involving
extenuating circumstances and procedural irregularities.
Other high stakes exams in the United States such as the
United States Medical Licensing Examination (USMLE)
focus only on a recheck of scoring and does not include a
manual review of the questions or candidate answers. In a
Google Scholar scan of the literature pertaining to factual
errors in other exams such as the MCCQE1 or the USMLE, I
am unable to find any literature or research regarding
concerns about factual errors in these exams. Given this
lack of comparator data, there is no way to know whether
the reports in this survey of between 11% and 40% of
questions containing factual errors are significant or
discrepant from candidate perceptions of other high stakes
entry to practice exams.
Also of concern are the many qualitative comments
suggesting that both CMGs and some IMGs have access to
past exam questions and that access to these past exam
questions provides an advantage to those in possession of
these questions in achieving success on RCPSC Exams.
While it is beyond the scope of this paper to verify this
claim, it is worth noting that the RCPSC has a strict
prohibition against sharing of copyrighted exam materials,
with language stating,
During registration at the exam site, each candidate
signs a statement confirming that they will respect the
confidentiality of the exam. If this confidentiality is
breached, exam results may be voided, and the
Canadian licensing authorities will be informed.17
The RCPSC website also states,
Exam questions are protected by copyright and are the
intellectual property of the Royal College of Physicians
and Surgeons of Canada. Any reproduction or other
disclosure of these exam questions, in whole or in part,
is strictly prohibited. Our Royal College will take all
available disciplinary measures and legal action
against any candidate or others who violate this
confidentiality provision.17
CANADIAN MEDICAL EDUCATION JOURNAL 2025
Despite the above prohibitions, of 37 responses from both
CMGs and IMGs to the question, “Did you have access to
past exam questions to help you study and prepare for the
RCPSC exam?,” 27 responded “Yes” and 10 responded “No”
with a total of 72.9% of those who answered indicating that
they had access to past exam questions. Fourteen
respondents answered this question by choosing “Not
Applicable.” It appears that both CMGs and IMGs had
access to past exam questions, although the qualitative
comments quoted above suggest that there may be
differential access to these past exam questions with IMGs
having reduced access relative to CMGs. Unfortunately, the
survey summary available online does not specify how
many of these 72% were CMGs and how many were IMGs.
This lack of a more in-depth analysis of data including cross
tabulations between questions is a limitation of this survey.
Blew and colleagues report that on past RCPSC
anesthesiology exams “Multiple-choice questions for each
examination are a 50/50 mixture of banked questions that
may have appeared previously on the written test and new
questions submitted by anesthesiologists at large and
edited by the Written Test Committee.”18(p804) This paper
dates from 2010 and hence this practice and the reported
percentages for utilization of banked questions may have
changed, however it is worth noting that on the SOCASMA
RCPSC Examination Experiences Survey 53.8% of
respondents reported finding that 31% or more of
questions appeared to be from past exams. If it is true that
a substantial number of RCPSC examination candidates
have access to past exams, and in particular CMG
candidates, this raises concerns regarding unfairness to
IMGs, particularly IMGs who have not trained in Canada, in
successfully completing RCPSC exams and gaining entry to
practice. This would be entirely inconsistent with the
principles of fairness outlined in the SEPT as they pertain to
“fairness in access to the constructs as measured.”
Transparent, objective, impartial
and fair: a legal standard for entry
to practice
Eight Canadian provinces (British Columbia, Alberta,
Saskatchewan, Manitoba, Ontario, New Brunswick, Nova
Scotia, and Newfoundland and Labrador) have legislation
that requires that regulatory Colleges assess qualifications
for entry to practice in a manner that is transparent,
objective, impartial and fair. Of these eight provinces, five
(Alberta, Saskatchewan, Manitoba, Ontario and
Newfoundland and Labrador) have language in fairness or
other legislation that requires that if a health regulator
relies on a third party to assess qualifications, it shall take
reasonable measures to ensure that the third party makes
the assessment in a way that is transparent, objective,
impartial and fair. Given this legislative framework which
recognizes the importance of an individual’s career and,
given that most MRAs require RCPSC Certification as a
condition of Registration, most MRAs have a duty to ensure
that the RCPSC examination and certifications are
transparent, objective, impartial and fair. It is of concern
that the above discussion raises questions regarding the
extent to which current RCPSC examination and
certification processes meet this standard.
Transparent
As discussed above, the Prat article and the SOCASMA
RCPSC Exam Experiences Survey both give rise to questions
regarding the extent to which RCPSC exam processes are
transparent. There appears to be a lack of public
information regarding details of exam construction,
validity, reliability, and outcomes; particularly outcomes
related to different cohorts such as CMGs relative to IMGs.
This lack of transparency is also reflected in concerns raised
by Prat, and in the SOCASMA RCPSC Exam Experiences
Survey regarding the creation and factual accuracy of
examination questions and the level of transparency the
RCPSC brings to this topic. While, as noted above, many
organizations such as MCC who employ high stakes
examinations do not permit formal reviews to focus on
incorrect information in their examinations, a more
transparent approach on the part of the RCPSC to issues of
examination development, validity, reliability, and
outcomes would likely have the effect of increasing public
and candidate confidence in the RCPSC examination
processes.
With respect to transparency, it appears that the RCPSC has
much of the information the public might desire to ensure
that RCPSC examination processes are objective and fair,
however the RCPSC lack of transparency regarding this
information means that the public does not have access to
this data. As discussed above, the RCPSC document “Roles
and Responsibilities of Examination Boards and Specialty
Committees with Regards to Examinations” references
many reports that the RCPSC compiles, including a
psychometric exam report, a MCQ report, and a post exam
survey report. Without access to these reports, it is
impossible to know exactly what they contain. It is possible
that having versions of these reports available to the public
would enhance transparency and increase public
confidence in the fairness of RCPSC examination processes.
CANADIAN MEDICAL EDUCATION JOURNAL 2025
Instead, the current lack of transparency regarding RCPSC
examination processes makes it extremely difficult to
conduct a critical review of RCPSC examination processes
and therefore almost impossible to accurately determine
the fairness of RCPSC examination experiences.
Objective
Without transparency regarding examination construction,
measures of validity, reliability, and outcomes, it is difficult
to evaluate the objectivity of RCPSC examinations.
Concerns have been raised by Prat and in the RCPSC
Examination Experiences Survey results regarding the
accuracy of some examination questions, as well as the
relevance of these questions to entry to practice. In the
interests of public confidence, it is important that
transparent and publicly available validity and reliability
research be undertaken to establish that RCPSC
examinations measure factors important to establishing
competence to enter independent practice, and not
esoteric knowledge or minutiae unique to the Canadian
medical context and culture. While an understanding of
Canadian medical culture and context is important, undue
focus on this creates bias and barriers which may be used
to favor domestic trainees, whereas the intended purpose
of the exam is reportedly to identify quality providers
based on their entry to practice level knowledge of safe,
effective practice in their specialty.
This point is made strongly in a recently released study of
systemic discrimination experienced by international
medical graduates.19 This study, using disaggregated data,
notes that barriers experienced by IMGs include lack of
transparency, and requirements that appear to be
designed to ensure exclusion. This study also reports it is
not unusual for an IMG doctor with five or ten years of
experience as a doctor or a professor in their specialty to
be equated to a recent Canadian graduate instead of being
recognized for their expertise and extensive training. In the
context of Canada’s current medical crisis and physician
shortage where IMGs are seen by some, including in a
recent Senate Report,20 as a significant part of the solution
to the physician shortage, a narrow focus on the Canadian
context vs. competence to treat patients can create
unnecessary barriers on physician recruitment and entry to
practice. There are better ways to address any deficiencies
in cultural knowledge than through entry to practice exams
such as brief cultural competency workshops. It may be
argued that an objective assessment of candidate
competency should focus predominantly on medical skills
and competence rather than on minutiae of Canadian
knowledge or the nuances of Canadian medical practice.
Impartial
The concerns raised by qualitative comments in the RCPSC
Exam Experiences Survey give rise to questions about the
impartiality of current RCPSC examinations. From the
comments in the SOCASMA RCPSC Exam Experiences
Survey, it appears to be an open secret that past
examination question banks exist, are critical to passing
RCPSC examinations, and are available to both CMGs and
IMGs studying and training in Canada, although they
appear not to be available to all IMGs, nor to IMGs who
trained outside Canada, and particularly Immigrant IMGs
without Canadian connections. Given Blew and colleagues
statement that in the past up to 50% of anesthesiology
examination questions were reused, and the reports from
the SOCASMA RCPSC Examination Experiences Survey that
31% or more of questions appeared to be from past exams,
differences in access to banks of past examination
questions would seem to have a high potential for creating
differential outcomes for CMGs and IMGs. In the absence
of transparent reporting by RCPSC of cohort differences in
outcomes between CMGs and IMGs including
differentiating between subgroups who did, and did not,
train in Canada, it is unclear whether IMGs as a cohort, and
particularly those without Canadian connections, are being
treated impartially, or whether this is an issue that needs
to be addressed to ensure an impartial and fair RCPSC
examination process.
Fair
Again, without transparency regarding test construction,
validity, reliability, and outcome reporting, it is difficult to
establish whether the RCPSC examination process meets
the standard of fairness. Certainly, the inequities identified
regarding differential access to past examination questions
for CMGs and IMGs raises questions regarding perceptions
of fairness. These are issues that should be of concern to
both the RCPSC and to MRAs who have the legal
responsibility to ensure transparent, objective, impartial
and fair registration processes, and fair access to the
profession for all Canadians. Regardless of their place of
education and place of training, Canadian citizens and
permanent residents who have demonstrated medical
competence should be entitled to fair access to the
profession of medicine.21
CANADIAN MEDICAL EDUCATION JOURNAL 2025
Context: factors that may affect
RCPSC ability to meet best
practice standards
The RCPSC website lists thirty specialties and thirty-seven
subspecialties that the RCPSC oversees.22 Each of these
specialties and subspecialties likely requires the creation of
a MCQ examination to assess candidates. This represents
an incredibly broad spectrum of responsibility for the
RCPSC in terms of examination development and
credential recognition. In comparison, the MCC is
responsible for the creation of only one high stakes MCQ
exam, making it much easier for them to thoroughly
research and establish the validity, reliability, and fairness
of their MCCQE1 exam. In contrast to the RCPSC’s
responsibility for some 67 specialty and subspecialty
exams, in the United States, there are 24 certifying Boards
organized under the American Board of Medical
Specialties23 with each Board responsible for the creation
of their own specialty and subspecialty examinations. This
is a much more achievable spectrum of responsibility.
Creation of objective, valid, reliable MCQ examinations is a
resource intensive undertaking involving subject matter
experts to create MCQ questions, oversight and exam
security considerations, scoring, and psychometric analysis
of examination results and score reporting. This process is
an expensive one to undertake, and to some degree costs
are passed on to certification candidates with estimated
costs (not including examination preparation costs) of
completing the RCPSC assessment and examination
certification process of $8,130 for CMGs and $12,045 for
IMGs.24 Given these costs and the challenges of developing
valid, reliable MCQ exams, it is possible that the RCPSC
utilizes reused multiple choice questions in an effort to
contain costs.
Regretfully, and based on the feedback from the SOCASMA
RCPSC Exam Experiences Survey, reusing questions
appears to have led to the creation of another problem,
which is the use by both CMGs and IMGs of question banks
made up of past exam questions that are still in circulation;
contrary to RCPSC confidentiality provisions. Based on the
RCPSC Exam Experiences Survey responses, there is a
perception that CMGs have ready access to these past
exams, while they are relatively unavailable to IMGs. This
may be even more of an issue for immigrant IMGs who
have fewer Canadian connections than Canadians studying
medicine abroad (CSA) and are less familiar with Canadian
culture and the nuances of the Canadian medical system.
Again, as the RCPSC Exam Experiences Survey did not
include an in-depth analysis of responses, it is not clear
how many IMGs and CMGs had access to past exams and
how this impacted their performance on exams.
Use of past sample questions is a common study tool, and
most medical students use question banks as a primary
study resource, however use of questions still in circulation
raises serious ethical and fairness issues. Such a practice
would be considered contrary to the rules of almost every
standardized exam. Official question banks are made from
old questions that are not still being used, although similar
ones might appear. Accessing questions that are still in use
as a study aid is highly problematic.
In the United States, standardized examinations such as the
United States Medical Licensing Exam have commercial
questions banks available from which candidates can study
such as UWorld. Many United States specialty licensing
Boards also have question banks available to candidates. In
Canada, as noted above, the MCC has practice exams
available for the MCCEQ1 and there are commercially
available question banks such as the Canada Q Bank.9 It
seems however, that the RCPSC has little available in the
way of study resources such as practice questions or
question banks for its various MCQ specialty and
subspecialty exams. Further, there appears to be a paucity
of commercially available question banks or study aids for
RCPSC examinations. The RCPSC does provide candidates
with a “Blueprint” or Exam Format document25 as a study
aid, but based on the RCPSC Exam Experiences Survey,
many candidates are either unaware of the “Blueprint”
documents, or do not find them helpful. Indeed, the RCPSC
Exam Experiences Survey almost all respondents indicated
they would find it helpful to have access to a bank of retired
exam questions, and having access to exam preparation
materials such as official question banks and review books
was prominent among the answers to the question, “What
other study resources would you find helpful to have
available?”
Graduated licensing, practice
ready assessments, and other
approaches
Recently, Thoma and colleagues have discussed the merits
of replacing high-stakes summative examinations with
graduated medical licensure.26 Thoma et al note that, “The
current medical licensure practices in Canada depend on
high-stakes, standardized, summative examinations that
were developed to uphold the medical social contract to
CANADIAN MEDICAL EDUCATION JOURNAL 2025
guard patient safety and benefit society. However, no
evidence has shown that these exams contribute to this
outcome.” They go on to state that, “the incredibly high
pass rates of graduates from Canadian training programs
suggest that this function is largely redundant.” Consistent
with the discussion in the section above, they also remark
that, “the exams remain painfully expensive for both the
trainee and the regulatory body, at a time when student
debt continues to soar.” They also note that, “The medical
licensing process directs valuable and limited resources
(e.g., time, energy, focus) toward passing an exam,
potentially detracting from the more meaningful goal of
preparing for independent practice.”
Instead, of high stakes exams, Thoma and colleagues
suggest a graduated licensing system where, “specialty-
specific examination boards would…be replaced with
boards trained to review standardized practice audits.”26
They further recommend that, “Credentialing and licensing
authorities should advocate for the resources required to
replace the current high-stakes summative assessments
with graduated licensure; to develop quantifiable,
nationally synergized, specialty-specific practice standards
that support both the oversight of graduated licensure and
maintenance of competence; and to streamline licensure
requirements between jurisdictions.”
A paradigmatic shift in credentialing such as that suggested
by Thoma and colleagues would offer a number of benefits
to organizations such as the RCPSC that appear to be facing
challenges in meeting best practice standards for
transparency in examination construction, validity,
reliability, and fairness. An approach consistent with that
advocated by Thoma and colleagues would be consistent
with the principles of Competence by Design developed by
the RCPSC in post graduate medical education programs
for CMGs. With regard to IMGs, the RCPSC is already
developing alternative streamlined methods of assessing
candidate competence such as Practice Ready Assessment
and their Practice Eligibility Route.27 Expanding these
pathways for IMGs and phasing out problematic high-
stakes examinations may help to reduce current perceived
barriers IMGs face in obtaining RCPSC Certification. Current
examination practices may be affected by fairness issues
for internationally trained candidates who come from
culturally diverse backgrounds that the current RCPSC
examination process may not accommodate in a fair and
equitable manner. Establishing these new pathways could
also assist in bringing the RCPSC into alignment with many
provincial MRAs who are actively seeking to reduce
barriers for IMGs so that they can contribute to solving
Canada’s current health care crisis.
Conclusion
The RCPSC is an institution that has accomplished much in
95 years and has much more to offer. It plays a critical role
in developing national competency standards to achieve its
vision of advancing learning for specialist physicians to
deliver the best health care for all. However, a critical
analysis of its examination and Certification practices raises
questions as to whether the RCPSC is meeting best practice
standards with respect to its broad mandate and usage of
high-stakes examinations. The RCPSC’s lack of
transparency regarding its examination processes and the
inaccessibility of data related to reliability and validity runs
the risk of generating mistrust on the part of examination
candidates, the public, and the medical profession
regarding the fairness and objectivity of the RCPSC’s
examination processes.
It may be time for the RCPSC and the MRA’s who delegate
the credentialing process to the RCPSC to reflect upon
RCPSC’s role as a credentialing organization and more
closely scrutinize RCPSC examination processes to ensure
they meet best practices and legislated standards of being
transparent, objective, impartial and fair. RCPSC, as an
independent non-profit corporation, has no statutory
duties or obligations, but as discussed above, the provincial
MRAs do have a legal duty to ensure that third party
assessors such as the RCPSC conduct their assessment in a
way that meets fairness standards. If third party assessors
fail to do so, these third parties such as the RCPSC must
either be encouraged to change their practices, or MRAs
need to find other methods of assessment that meet
legislative standards of fairness. Such other methods might
involve bringing specialty assessment in house with MRAs
doing their own assessments of specialty qualifications.
Particularly in a changing environment marked by global
mobility, RCPSC's traditional exam-based approach to
certification may have become outdated. The exam
process is both resource intensive in terms of creation and
validation of exams and is increasingly time-consuming and
expensive for both the RCPSC and candidates in an
environment of fiscal restraint. Perhaps, particularly as
Competence Based medical education is established for
Canadian medical graduates, and as we recognize the
validity of various internationally recognized approaches to
medical training that result in competent practitioners in
other countries, it is time for the RCPSC to move toward
alternate methods of assessment suggested by other
CANADIAN MEDICAL EDUCATION JOURNAL 2025
authors. Such approaches may include graduated medical
licenses, Practice Ready Assessments, and maybe a
portfolio approach to assessment of candidate credentials
and competence in conjunction with periods of supervised
practice.
Conflicts of Interest: The Author is a volunteer with SOCASMA,
providing strategic guidance on topics related to this paper. He has
engaged in dialogue with the RCPSC, the subject of this paper,
regarding issues raised herein, and was part of the SOCASMA team
that conducted the survey involving examination experiences
referenced in this paper.
Funding: The author has no financial affiliations to declare.
Edited by: Marcel D’Eon (editor-in-chief)
References
1. Royal College of Physicians and Surgeons of Canada.
About/History. Available from
https://www.royalcollege.ca/en/about/history [Accessed Jul 5,
2024]
2. Royal College of Physicians and Surgeons of Canada.
Competence by Design. Available from
https://www.collegeroyal.ca/en/standards-and-
accreditation/competence-by-design.html [Accessed Jul 5,
2024]
3. Royal College of Physicians and Surgeons of Canada. Standards
documents. Available from
https://news.royalcollege.ca/en/standards-and-
accreditation/pgme-accreditation [Accessed Jul 5, 2024.
4. American Educational Research Association, the American
Psychological Association, and the National Council on
Measurement in Education. Standards for psychological and
educational testing. 2014. Available from
https://www.testingstandards.net/uploads/7/6/6/4/76643089
/standards_2014edition.pdf [Accessed Jul 5, 2024]
5. Tavakol M, Stewart C, Sharpe, CC. Ensuring fairness in
assessment in health professions education: rapid analysis
tools to detect differential item functioning across groups.
IJME, 15:80-83 https://doi.org/10.5116/ijme.6694.de69.
6. Medical Council of Canada. MCCQE Part 1. Available from
https://mcc.ca/examinations-assessments/mccqe-part-i/
[Accessed Jul 5, 2024]
7. Medical Council of Canada. MCCQE Part 1 annual technical
report 2020. Available from https://mcc.ca/wp-
content/uploads/MCCQE-Part-I_Technical-Report_2020.pdf
[Accessed Jul 5, 2024]
8. Medical Council of Canada. Preparatory products. Available
from https://mcc.ca/examinations-assessments/resources-to-
help-with-exam-prep/preparatory-products/ [Accessed Jul 5,
2024]
9. Canada Q Bank. QBank for the MCCQE Part 1. Available from
https://www.canadaqbank.com/mccqe-part1.php [Accessed
Jul 5, 2024]
10. Royal College of Physicians and Surgeons of Canada.
Developing multiple choice questions for the RCPSC certification
examinations. 2004. Available from
https://cmapsconverted.ihmc.us/rid=1P2XTLCSV-19DYZD-
BDK/Wood_et_coll_2004_RCPSC_MCQ_guidelines.pdf
[Accessed Jul 6, 2024]
11. Royal College of Physicians and Surgeons of Canada. Roles and
responsibilities of examination boards and specialty
committees with regards to examinations. Available from
https://www.royalcollege.ca/content/dam/documents/accredi
tation/specialties/member-resources/specialty-
committee/roles-responsibilities-of-examination-specialty-
committees-regards-to-examinations-e.pdf [Accessed Jul 6,
2024]
12. Royal College of Physicians and Surgeons of Canada. Average
pass rates by specialty or subspecialty. Available from
https://www.collegeroyal.ca/en/eligibility-and-exams/exam-
results/average-pass-rates [Accessed Jul 6, 2024]
13. Prat, S. Who is monitoring the gatekeepers? The Royal College
an impenetrable fortress. Oct 23, 2022. Available from
https://healthydebate.ca/2022/10/topic/royal-college-exam/
[Accessed Jul 6, 2024]
14. Royal College of Physicians and Surgeons of Canada. Policy on
formal review of examinations Available from
file:///C:/Users/Owner/OneDrive/Documents/Policy%20on%20
Formal%20Reviews%20and%20Appendix.pdf [Accessed Jul 6,
2024]
15. Society for Canadians Studying Medicine Abroad (SOCASMA).
Royal college of physicians and surgeons exam experiences
survey summary. Available from https://socasma.com/wp-
content/uploads/2023/11/RCPSC-EXAM-EXPERIENCES-SURVEY-
SUMMARY.pdf [Accessed Jul 6, 2024]
16. Government of Canada. Panel on research ethics. Available
from https://ethics.gc.ca/eng/tcps2-eptc2_2022_chapter2-
chapitre2.html [Accessed Jul 6, 2024]
17. Royal College of Physicians and Surgeons of Canada. Exam
process overview. Available from
https://www.royalcollege.ca/en/eligibility-and-exams/exam-
process [Accessed Jul 9, 2024]
18. Blew P, Muir JG, Naik VN. The evolving Royal College
examination in anesthesiology. Can J Anesthes. 57, 2010, Pg
804-810. https://doi.org/10.1007/s12630-010-9341-1
19. Mosaic. Using disaggregated data to address the systemic
discrimination experienced by international medical graduates.
May 2024. Available from https://mosaicbc.org/international-
medical-graduates/ [Accessed Nov 18, 2024]
20. Kutcher S, Ravalia M, Omidvar R, El-Assal K. Maximizing
medical talent: how Canada can increase the supply of family
doctors by 50% quickly and cost effectively. Senate Report, Oct
2024. Available from https://www.ratnaomidvar.ca/wp-
content/uploads/2024/10/IMG-Report-Design-EN.pdf
[Accessed Nov 18, 2024]
21. MacFarlane, M. When a Canadian is not a Canadian:
Marginalization of IMGs in the CaRMS Match. Can Med Ed J.
12(4), 2021, Pg 132-140. https://doi.org/10.36834/cmej.71790
22. Royal College of Physicians and Surgeons of Canada.
Information by discipline. Available from
https://www.royalcollege.ca/en/standards-and-
accreditation/information-by-discipline [Accessed Jul 6, 2024]
23. American Board of Medical Specialties. Available from
https://www.abms.org/ [Accessed Jul 6, 2024]
24. Royal College of Physicians and Surgeons of Canada.
Assessment and exam fees. Available from
CANADIAN MEDICAL EDUCATION JOURNAL 2025
https://www.royalcollege.ca/en/eligibility-and-exams/exam-
process/assessment-and-exam-fees#assessment [Accessed Jul
6, 2024]
25. Royal College of Physicians and Surgeons of Canada. Prepare
for your Royal College Exams. Available from
https://www.collegeroyal.ca/en/eligibility-and-exams/exam-
preparation [Accessed Jul 6, 2034]
26. Thoma B, Monteiro S, Pardhan A, et al. Replacing high-stakes
summative examinations with graduated medical licensure in
Canada. Can Med Assoc J. 194(5) 2022.
https://doi.org/10.1503/cmaj.211816
27. Royal College of Physicians and Surgeons of Canada. What are
we doing about Canada’s health human resource crisis?
Available from
https://news.royalcollege.ca/en/newsroom/posts/what-are-
we-doing-about-canada-s-health-human-resource-crisis-.html
[Accessed Jul 6, 2024]
