[guidelines workshop * atelier sur les guides de pratique ]
REPORT ON ACTIVITIES AND ATTITUDES
OF ORGANIZATIONS ACTIVE
IN THE CLINICAL PRACTICE GUIDELINES FIELD
Anne 0. Carter, MD, MHSc, FRCPC; Renaldo N. Battista, MD, ScD, FRCPC;
Matthew J. Hodge, BA; Steven Lewis, MA; Antoni Basinski, MD, PhD; David Davis, MD, CCFP, FCFP
The organizingcommittee of aworkshopon clinicalpractice guidelines (CPGs) surveyed invited orga-
nizations on their attitudes and activities related to fivetopicsto be coveredduringthe workshop ses-
sions: organizational roles, priority setting, guidelines implementation, guidelines evaluation and devel-
opment of a network of those active in the CPG field. Organizational roles: The national specialty
societies were felt to have the largest role to play; the smallest roles were assigned to consumers, who
were seen to have a role mainly in priority setting, and to industry and government, both of which
were seen to have primarily a funding role. Many barriers to collaboration were identified, the solu-
tions to all of which appeared to be better communication, establishment of common principles and
clear role definitions. Priority setting: There was considerable agreement on the criteria that should be
used to set priorities for CPG activities: the burden of disease on population health, the state of scien-
tificknowledge, the cost of treatment and the economic burden of disease on society were seen as im-
portant factors, whereas the costs ofguidelines development and practitioner interest in guidelines de-
velopment were seen as less important. Organizations were unable to give much information on how
theysetpriorities. Guidelines implementation: Most of the organizations surveyed did not actively try to en-
sure the implementation of guidelines, although a considerable minority devoted resources to imple-
mentation. The 38% of organizations that implemented guidelines actively listed a wide variety of ac-
tivities, including training, use of local opinion leaders, information technology, local consensus
processes and counter detailing. Guidelines evaluation Formal evaluation of guidelines was undertaken by
fewer than 13% of the responding organizations. All the evaluations incorporated assessments before
and after guideline implementation, and some used primary patient data. Barriers to evaluation in-
cluded lack ofmoney, time, data or expertise. CPG network: Most of the respondents felt that all organi-
zations and individuals interested or involved in guidelines should form the membership of the net-
work. The three most important functions of such a network were deemed to be (a) to facilitate
collaboration among those involved in the CPG process, (b) to maintain an information centre on
CPGs and (c) to provide expertise to the CPG process. It was felt that the network should have some
formal structure and communicate through e-mail and print media.
Le Comite organisateur d'un atelier sur les guides de pratique clinique (GPC) a effectue, aupres des or-
ganisations invitees, un sondage sur leurs attitudes et leurs activites face a cinq sujets qui seraient abor-
des au cours des ateliers: roles organisationnels, etablissement de priorites, mise en oeuvre des guides,
evaluation des guides et creation d'un reseau d'intervenants actifs dans le domaine des GPC. R6les orga-
nisationnels: Les representants croyaient que les societes nationales de specialites ont le role le plus im-
Dr Carter is associatedirector, Department ofHealth Care and Promotion, CMA, Ottawa, Ont. Dr Battista is professor in the departments of Epidemiology and Biostatistics, Family Medicine and
Medicine, McGillUniversity, Montreal, Que. Mr. Hodge is an MD-PhD candidate in the Faculty ofMedicine, McGill University. Mr Lewis is the chief executive officer of the Saskatchewan Health
Services Utilization and ResearchCommission, Saskatoon, Sask. Dr Basinski is seniorscientist at the Institute for Clinical Evaluative Studies in Ontario, Toronto, Ont. Dr. Davis is associate dean,
Continuing Education, FacultyofMedicine, McMaster University, Hamilton, Ont.
This article is the first in a series
ofsixto appearin theOctober,November and December issues ofCMAJ.
O.Carter, Departmentof Health Care andPromotion, CanadianMedicalAssociation, P0 Box8650,Ottawa ON
*- For prescribing information see page 1006
<-- Forprescribing information seepage1006
CAN MED ASSOC J*OCT. 1, 1995; 153 (7)
portant 'a jouer et que les roles les moins importants reviennent aux consommateurs qui doivent inter-
venir surtout dans l'etablissement des priorites, et a l'industrie et au secteurpublic, auxquels ont at-
tribue un role de financement avant tout. On a defini de nombreux obstacles 'a la collaboration, dont
le'limination semble reposer dans tous les cas sur une meilleure communication, surl'etablissement de
principes communs et sur la definition d'un role clair.gtablissementde priorites
portant sur les criteres qu'il faudrait utiliser pour etablir les priorites relatives aux activites portant sur
les GPC: le fardeau que les maladies imposent a la sante des populations,F6tatdes connaissances
scientifiques, le cout des traitements et le fardeau financier que la maladie impose 'a la societe ont ete
consideres comme des facteurs importants, tandis que les couts d'elaboration des guides etl'interet des
praticiens 'a cet egard a ete juge moins important. Les organisations n'ont pu fournir beaucoup de ren-
seignements sur fa facon dont elles sty prennent pour etablir les priorites. Mise en oeuvre des guides: La plu-
part des organisations interrogees n'essayaient pas activement dassurer la mise en oeuvre des guides,
meme si une minorite importante y consacrait des ressources. Plus de 38 % des organisations qui ont
mis en oeuvre des guides ont enumere un large eventail d'activites: formation, recours 'a des dirigeants
d'opinion locaux, technologie de linformation, processus de consensualisation 'a lIechelle locale et
description detaillee au comptoir. .valuation des guides: Moins de 13 % des organisations qui ont
repondu procedaient 'a une evaluation structur&e des guides. Toutes les evaluations comportaient une
evaluation avant et apres Ia mise en oeuvre du guide et certaines utilisaient des donnees primaires sur
les patients. Les obstacles 'a lIevaluation comprenaient le manque d'argent, de temps, de donnees ou de
comp'tences specialisees. R'seau de GPC: La plupart des repondants etaient davis que le r'seau devrait
etre constitue de toutes les organisations et les personnes qui s'occupent de guides ou qui sty
interessent. On a juge que les trois fonctions les plus importantes d'un tel reseau seraient les suivantes:
a) faciliter la collaboration entre les intervenants du processus d'elaboration des GPC, b) maintenir un
centre dinformation sur les GPC et c) fournir des competences specialisees au processus d'elaboration
des GPC. Les repondants croyaient que les reseaux devraient avoir une structure officielle et communi-
quer 'lectroniquement et par ecrit.
11 y avait un consensus im-
At a November 1992 workshop involving individuals
and organizations interested in clinical practice
guidelines (CPGs), participants agreed that it would be
valuable to hold future workshops on specific topics in
the guidelines field.-7 An advisory committee was orga-
nized, comprising representatives of the main stake-
holder and funding groups, and another workshop was
planned for the fall of 1994 to focus on four of the most
difficult issues in the guidelines field: organizational
roles, setting of priorities, implementation of guidelines
and evaluation. In addition, the committee decided to
hold a session to build on the concept, formed at the
first workshop, of a network to support and promote
guidelines activities. It felt that all five issues would be
best dealt with if organizations sending representatives
to the workshop were surveyedon their activities and at-
titudes with regard to these issues. Accordingly, those
charged with developing each of the issue sessions de-
signed a survey that was sent to invited organizations.
This report summarizes the findingsof thesurvey.
All organizations invited to the workshopwere sent a
questionnaire soliciting their views on the topics to be
covered: organizational roles, priority setting, imple-
mentation, evaluation, and terms of reference for a CPG
network. Invitees consisted of all the organizationsiden-
tified by the CMA Quality of CareProgramasbeingac-
tively involved in or considering active involvement in
the CPG field at the national or provincial/territorial
level. This identification process involved extensive con-
sultation with and solicitation of information from na-
tional and provincial organizations for more than 2
years. No effort was made to identify locally active orga-
nizations. Although new organizations were continually
being identified through this process, those identified af-
ter July 1, 1994, were sent an invitation to the workshop
but were not sent the questionnaire because there was
insufficient time to incorporate their responses into the
It was left up to each organization to determine how
and by whom the questionnaire would be completed.
Organizations were asked to mail completed question-
naires to the CMA. Nonresponding organizations were
telephoned close to the response deadline and reminded
that the questionnaire was due. Questionnaire responses
were analysed in a descriptive manneronly.
Questionnaires were sent to 107 organizations along
with an invitation to nominate someone to attend the
workshop. By the deadline for analysis 55 completed
questionnaires (51%) had been received from the follow-
ing categories of organization: 8 (62%) of 13 licensing
authorities, 12 (71%) of 17 governments orparagovern-
mental organizations, 6 (43%) of 14provincialand terri-
torial medical associations or other organizations repre-
senting physicians provincially, 18 (51%) of 35 national
CAN MED ASSOC J * ler OCT. 1995; 153 (7)
specialty societies and 11(39%) of 28 other types of or-
ganizations. Since the respondents could indicate several
answers to many of the questions, the number of
sponses to those items often exceeded 55. In addition,
some of the respondents did not answer some questions,
so that the number of responses at times was less than 55.
Of all the respondents
10 (180/) were not currently
involved in guidelines activities. These were mainly from
specialty societies. Of the remaining 45 respondents
(40%) were involved in funding activities, 25 (56%)
priority setting, 24 (53%) in guidelines development, 34
in guidelines dissemination, 20 (44%)
15 (33%) in guidelines evaluation,
in coordination of guidelines activities,
(3 3%) in endorsing guidelines, 11 (24%) in teaching and
9 (20%) in research involving guidelines. Most organiza-
tions were involved in several different typ'es of activi-
ties. The following sections summarize the responses to
the five issues covered by the questionnaire.
Attitudes toward the appropriate roles oforganizations
The respondents completed a matrix indicating their
opinions about the appropriate level of involvement (es-
sential, desirable or should not participate) of various
types of organization
in each aspect of CPG activity,
from funding to research.
responses were as-
signed a score of 2,
a score of
a score of 0. There
was no differentiation in- the scoring of the last two re-
sponse types: some respondents left several cells in the
matrix blank, and thus the definition of the zero score
had to be expanded. Although doubtless some
responses were implicit in the blank cells,
generally assumed that respondents would have identi-
fied positive roles. (The complete set of data is available
from the authors upon request.)
All of the respondents indicated that specialty soci-
eties should have the most involvement in guidelines ac-
tivities and that industry should have the least (Table 1).
a moderate endorsement of involvement
in establishing priorities for CPGs (i.e., choosing sub-
jects), the respondents saw little role for consumers
any CPG-related activities.
Most of the respondents indicated that there are na-
tional, provincial and local roles in CPG activities; sup-
port for provincial involvement was virtually unanimous.
Setting priorities and guidelines development were con-
sidered to be national responsibilities by some, provin-
cial by others and joint by several. Almost all of the re-
spondents identified local roles in implementation.
Barriers to collaboration
The respondents were asked to indicate the barriers
to collaboration in the CPG process. Responses varied,
but the following main themes emerged.
The medical community is uncertain whether CPGs
are necessary and helpful, or whether they may com-
promise autonomy and clinical judgement.
There is a strong perception of territoriality and turf
protection that stands in the way of national coordi-
nation and development efforts.
Some perceive that traditional medical practices and
attitudes are resistant to the concept of CPGs, which
are designed to reduce variations in practice.
'C PG activity
1.0 J# .,7
. .0$ 09
- -.5- 122 .0
CAN MED ASSOC J *OCT.
1, 1995; 153 (7)
1... L, 1.
l,' :,. Ir
4. There is no common understanding of which organi-
zations should be involved in which aspects of CPG
5. Different organizations and jurisdictions may have
legitimately different interests and priorities, creat-
ing difficulties for national coordination and priority
Breaking down barriers to collaboration
The main solutions proposed included better commu-
nication, a set of national principles for CPGs, clearer
role identification, more stable funding and a clear focus
on evidence-based development.
Criteria for setting priorities
On a 5-point scale from "very important" (1) to "not
at all important" (5) the respondents were asked to rate
the relevance of each of seven criteria: the health burden
on the population, the economic burden of disease on
society, the cost of treatment to the health care system,
the extent of practice variation, the state of scientific
knowledge, the cost of guidelines development and
practitioner interest in having guidelines developed. The
respondents were not required to rank the criteria and
could thus rate all seven the same if they wished.
For each category of organization, average scores
for each item were calculated. Between-group differ-
ences were not significantly different. Table 2 presents
the criteria that were deemed very important or impor-
tant, by category. Very important items had average
scores of less than 2, and important items had average
scores of 2 or more. No item received an average score
greater than 3.2.
We found no significant or meaningful differences be-
tween the priorities of different stakeholders. All of the
groups identified the health burden on the population as
a very important criterion for setting priorities for CPG
activities. Similarly, all of them rated the costs of guide-
lines development and practitioner interest as being rela-
tively less important.
Methods for setting priorities
Details about how priorities are set proved elusive.
The respondents generally described who was in charge
of setting priorities but omitted information about how
this was done. Most described some form of reactive pri-
ority setting rather than a proactive canvassing of mem-
bership, the public or other groups. When asked to
identify groups consulted during priority setting, the re-
spondents most often identified members of their re-
spective organizations and other professional or spe-
cialty societies. Community members were identified
least frequently, by only 8 (32%) of 25 respondents.
Licensing body (n =8)
Health burden on population
Economic burden on society
Costs of tretment
State of scientific knowledge
Health burden on population
Economic burden on society
Cost of treatment
Health burden onpopulation
State of scientific knowledge
Practitioner interstd in guideiines
.......Health.;-1~dn on p!.p..ti!on
Cost ofguidelines development
Practitioner interest in guidelines:
(n - 12)
State of scientific knowledge
Cost of guidelinesdevelopMent
Practitioner interest in gutdelines
Economic burden on society
Cost of treatment
Cost of guiderines develpment
association or organization
representing physicians at
provincial level (n =6)
. Natioxnal specialty society.
CoK of treatmet
tif guideline d
P tner inte..rest i pi
Eco*o'c br'iv-l7 Eciety
State o scientific knowledge
- - -: :ati
CAN MED ASSOC J * i11 OCT. 1995; 153 (7)
DiSSEMINATION AND IMPLEMENTATION
Most (62% [34/55]) of the respondents indicated that
their organizations disseminated CPGs. Of these, 82%
(28/34) disseminated the CPGs developed by their own
organizations; 47% (16/34) disseminated guidelines de-
veloped by other organizations.
Active implementation activities
Most (58% [32/55]) of the respondents indicated that
they did not actively try to ensure the implementation of
guidelines disseminated by theirorganization; 21 (38%) of
the 55 indicated that they did try, particularly if the CPGs
had been developed by that organization (17 [81 %]).
Fourteen (25%) of the 55 respondents indicated that
staff resources were used to disseminate and implement
CPGs, ranging from small percentages of a full-time equiv-
alent to three full-time equivalents. Twenty (36%) reported
that dissemination and implementation of CPGs were the
responsibility of committees, of which 75% were standing
committees and 50% ad-hoc committees. Finally, 10
(18%) of the respondents indicated that their organization
gave financial support- in one instance up to $500 000 a
year-to the dissemination and implementation of CPGs.
Specific dissemination and implementation activities
Twenty-one (38%) of the 55 respondents indicated
their involvement in the following activities, listed in or-
der of frequency.
Direct mailing of CPGs to others (88%).
Direct mailing of CPGs to members (85%).
Publication of CPGs in journals or newsletters (85%).
Organization of conferences or workshops (74%).
Sponsorship of conferences or workshops (64%).
Sponsorship of research into the dissemination and
implementation of CPGs (49%).
Training and support of influential educational lead-
ers (local opinion leaders) (44%).
Publicizing CPGs to patients or the public (34%).
Use of computer technology (34%).
Sponsorship of local consensus processes around
centrally developed CPGs (28%).
Training and support for audit and feedback or
prompting (reminders) (26%).
Face-to-face visits (counter detailing or outreach vis-
Integration into recertification or licensing examina-
Promotion of CPGs inpeer reviews (less than 5%).
Use of audiovisual materials (less than 5%).
EVALUATION OF GUIDELINES
Formal evaluation of CPGs is rare. The respondents
were asked to identify any guidelines that had been for-
mally evaluatedby their organization and the evaluation
design. Only seven organizations had formally evaluated
CPGs or weredoing so, all since 1992. Two of the orga-
nizations were provincial medical associations, two were
cancer agencies, one was a provincial college of physi-
cians and surgeons, and two were health services or re-
search organizations. All of the evaluations incorporated
assessments done before and after dissemination of the
CPGs and were based on administrative data such as lab-
oratory test volume. The hospital, cancer agencies and
one provincial medical association also includedprimary
patient data in their assessments. The types of CPGs
Only one organization claimed that it had madeany
specific changes to CPGs or to its activities on the basis
of the evaluation results.
Main purpose of evaluation
Six of the seven organizations currently active in
CPG evaluation cited some aspect of quality of care as
the main reason for evaluating CPGs. The general thrust
of the responses was that the most appropriate and cost-
effective care should be provided through a monitoring
of outcomes during quality-assessment activities.
Barriers to evaluation
All of the respondents were asked what they consid-
ered to be major barriers to the evaluation of CPGs. The
most common barrier cited was lack of money or re-
sources (36% [20/55]). The next commonest were lack of
time (18% [10/55]), of data or systems (18% [10/55]) and
of the organization's expertise (16% [9/55]). Other barri-
ers included lack of clear goals and objectives, lack of for-
malized processes, lack of commitment, difficulty relating
outcome to the intervention and fear of criticism.
Major supports for evaluation
Major supports for the evaluation of CPGs were var-
ied and difficult to categorize. In general, the themes
CAN MED ASSOC J * OCT. 1, 1995; 153 (7)